StreamBase Engine Configuration

Overview

This article provides a reference for writing a StreamBase engine configuration file where the HOCON type is com.tibco.ep.streambase.configuration.sbengine.

The root objects in this configuration type are:

StreamBaseEngine

This root object contains data such as custom plug-in function definitions and module definitions. When the StreamBase engine starts, information in this configuration is combined with information from all the separate configuration types to create a consistent configuration for its StreamBase server instance to use.

EventFlowDeployment

This root object can contain modules, container connections, and operator parameters that can be set independently of the rest of a StreamBase engine's configuration.

ParameterSettingsGroup

An EventFlow parameter settings group is an associative array of parameter settings objects, indexed by name.

CustomFunctionGroup

An EventFlow custom function group is an associative array of custom function definitions, indexed by name.

Required Header Lines

Each configuration file must contain the following header lines, typically found at the beginning of each file:

name

Specifies an arbitrary, case-sensitive string to name this configuration, which must be unique among other files with the same type, if any. Configuration files can refer to each other by this name. Select a name that reminds you of this configuration's type and purpose. For example:

name = "mysbengine"
version

Specifies an arbitrary version number that you can use to keep track of file versions for this configuration type in your development project. The maintenance of version numbers is under user control; StreamBase does not compare versions when loading configuration files during the fragment launch process. The version number is a string value, and can contain any combination of characters and numbers. For example:

version = "1.0.0"
type

This essential setting specifies the unique HOCON configuration type described on this page.

type = "com.tibco.ep.streambase.configuration.sbengine"

The three header lines taken together constitute a unique signature for each HOCON file in a project's configurations folder. Each project's configurations folder can contain only one file with the same signature.

The top-level configuration object defines the configuration envelope the same way for all HOCON file types.

configuration

On a line below the header lines, enter the word configuration followed by an open brace. The configuration object is a sibling of the name, version, and type identifiers, and serves to define the configuration envelope around this type's objects as described on this page. The file must end with the matching close brace.

configuration = {
...
...
}

StreamBaseEngine Root Object

This section shows the configuration of the objects and properties that are valid for the StreamBaseEngine root object. StreamBaseEngine is used to configure the behavior of an individual engine that runs within a StreamBase Runtime node. See the sample configuration in Full StreamBaseEngine Example.

StreamBaseEngine

This root object describes a top-level StreamBase engine configuration. The StreamBase engine configuration inherits from the base Runtime Java engine.

associatedWithEngines

If you want to restrict this object to be associated with specific engines, do so here. Each entry can be a specific engine name or a regular expression that can applies to more than one engine. This array is optional and has no default value. If not present, the configuration is associated with all engines.

For example:

associatedWithEngines = [ "javaengine", "otherengine[0-9]" ]
jvmArgs

String. A array of JVM arguments used for this engine instance on this operating system type. These values can be overridden by individual application fragments. This array is optional and has no default value.

Do not set the classpath here. Use the guidance in Java VM Memory Settings for assistance when setting the -Xms (minimum) and -Xmx (maximum) Java heap size arguments. While it is possible to specify system properties here, it is better to use the systemProperties property described below.

For example:

jvmArgs = [
  "-Xmx3g"
  "-Xms512m"
  "-XX:+UseG1GC"
  "-XX:MaxGCPauseMillis=500"
  "-XX:ConcGCThreads=1"
  ]
systemProperties

A list of Java system properties that will be set in each engine instance on this operating system type. These values can be overridden and extended by individual application fragments. This object is optional and has no default value.

System properties are represented as a HOCON object, where each property name in the object is a system property name and each property value is a system property value. HOCON objects are delimited by braces.

For example:

systemProperties = { 
  name1 = "value1" 
  "name.2" = "value2" }

For the second example above, note the dot, which is a path separator and therefore must be inside the quoted string.

externalClassPath

String. Specifies a list of JAR files and class hierarchies to add to the application fragment's classpath. This array is intended for fragments that use common external Java libraries. Paths are required to be absolute and in Java, forward-slash format, even on Windows. This array is optional and has no default value.

The third entry in this example array is a path to a folder containing a class file hierarchy:

externalClassPath = [
  "/absolute/path/d.jar"
  "/absolute/path/g.jar"
  "/absolute/path/root"
]
jvmTuning

Specifies various tuning parameters to pass to the Java virtual machine that runs the StreamBase server. This object is optional and has default values as described below.

minimumDispatchThreads

Minimum number of dispatch threads. This property is optional and its default value is 10.

For example:

minimumDispatchThreads = 10
maximumDispatchThreads

Long. Specifies the maximum number of dispatch threads. This property is optional and its default value is 2000. For example:

maximumDispatchThreads = 2000
shutdownTimerSeconds

Long. Specifies the maximum number of seconds to wait for a JVM to shut down before aborting it. This property is optional and its default value is 60. For example:

shutdownTimerSeconds = 60
timerParallelism

Long. Specifies the number of timers that can be running concurrently. This property is optional and its default value is 1. For example:

timerParallelism = 1
timerResolutionMilliseconds

Long. Specifies the maximum timer resolution. This is the interval at which timers are examined. Higher resolution timers have more impact on system performance. This property is optional and its default value is 1000. For example:

timerResolutionMilliseconds = 1000
schedulerPolicy

The scheduling policy for the JVM process. The valid values for the policy are SCHED_FIFO, SCHED_RR, and SCHED_OTHER. This property is optional. For example:

schedulerPolicy = "SCHED_FIFO"
schedulerPriority

Long. Specifies the scheduling priority for the JVM process. The valid range for priority depends on the policy; for Linux the valid values for SCHED_FIFO and SCHED_RR are 1 to 99. This property is optional and its default value is operating system-specific. For example:

schedulerPriority = 1
streamBase

StreamBase EventFlow-specific configuration. The EventFlow-specific information is contained in its own sub-configuration so it does not collide with any LiveView information (where a LiveView engine configuration extends the EventFlow one).

dataAreaPath

String. A path to an engine-specific data directory, which Spotfire Streaming operators, adapters, and other services use to store file data. The path can be absolute or relative; relative paths are relative to the value of the default engine data area location.

This path is created if it does not exist. If the path cannot be created or accessed, configuration audit fails. The data area can be changed at runtime by replacing a configuration with a new path. This location is never removed even if the data area is changed by replacing configuration, or the configuration is deactivated and removed.

This property is optional and has a default value of engine-data-area.

For example:

dataAreaPath = "engine-data-area"

The default folder name is created by default in the current node's node directory, in nodename/fragments/enginename/engine-data-area. See the configuration file in the Data Directory sample for examples of settings relative to the default path in Studio, and of absolute paths.

operatorParameters

String. An association of global parameters to use in expressions in any module or container run by this StreamBase engine. This object is optional and has no default value. Operator parameters are legacy global variables, superseded by module parameters.

An operator parameter is specified by a name and a value. The name can be a simple name such as ParamOne. This will:

1. Expose a parameter available to modules using ${param} syntax. (This syntax also resolves to module parameters defined directly in modules, when present.)

2. Set the given value as the parameter value for any Java operator or adapter with a parameter of the given name.

The name can also be a dotted path name to set a particular adapter or operator's parameter, such as ContainerName.OperatorName.ParamOne. If the operator is in a submodule you must include the submodule in the dotted pathname, such as ContainerName.SubModuleName.OperatorName.ParamOne. When a path is used the parameter value will only be set for that named operator. All other parameters with the same name on other operators remain unmodified. See String Values in Parameters in the Authoring Guide for the recommended policy on quoting string parameter values.

This optional property is encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.

For example:

MyName = "MyValue"
"MyName.with.dots.quoted" = "#!v0hZ40tTIC1PKW8TrYdqyW81KLo2kH
   JG/GaYZ/K+1Z/UrhrlG1nfMFfuCJY+3Us4Vvt6jyAwiRcLk6aZN3vUXg=="
forceHighResolutionTimer

Bool. Set to true to force high resolution timers, at the expense of performance. This property is optional and its default value is false.

For example:

forceHighResolutionTimer = true
ignoreUnboundCaptures

Bool. StreamBase's default behavior is to fail typechecking when an unbound capture is detected. An unbound capture occurs when an input stream of a module is implemented with a capture field and that input stream is not connected to an input port of the calling Module Reference in the parent application. This situation results in an error because there is no schema to examine or fields to capture for the unbound capture instance. When you are certain your unbound captures are benign, you can optionally suppress the resulting typecheck errors. To do this, set this key to true. This property is optional and its default value is false.

schemaMaxSize

Long. The maximum size of a stream's schema; that is, the total number of bytes of all properties you expect in the tuple. While there is no theoretical upper limit, in practice there is a maximum based on the available memory on the machines that will host the processing of your application. Be cautious about creating very large schemas, because applications that move smaller amounts of data perform much better than applications that move tuples containing unnecessary fields. This property is optional and its default value is 1 megabyte.

For example:

schemaMaxSize = 1M
nowImplementation

What implementation should be used for calls to now(). Must be either system or thread.

system uses a call to Java's System.currentTimeMillis(). thread uses a background thread that checks the time approximately every millisecond. This option results in decreased accuracy, but may be more efficient than system if you call now() more frequently than 1000 times per second. This property is optional and its default value is system.

For example:

nowImplementation = "thread"
operatorStateChangeTimeoutMilliseconds

Int. Each Java Operator changes state along with the engine process as a whole. The engine waits for each Operator to change state before it completes its state change. The value of this property is the amount of time the engine will wait before timing out the Operator. If an operator is timed out on a state change, the engine shuts the operator down and proceeds with the state change. This property is optional and its default value is 10000.

For example:

operatorStateChangeTimeoutMilliseconds = 10002
parallelRegionQueues

Engine-wide configuration settings for parallel region queues.

defaultDataDistributionPolicy

The data distribution policy to use for parallel region queues. This property is only valid if the parallel region in question uses transactional memory for storage, in which case it enables high availability of that storage. Its value must be a valid data distribution policy as declared in your application configuration. This property is optional and has no default value. If it is not set, then high availability for this region's queues is not enabled.

defaultInitialBufferSize

The initial buffer size for a parallel region queue to use. Must be a power of 2. These buffers can grow, so setting this is not technically necessary. This property is optional and its default value is 16.

For example:

defaultInitialBufferSize = 16
defaultMaxBufferSize

The maximum queue size in tuples for a parallel region queue. This must be a power of 2. This property is optional and its default value is 2^30.

For example:

defaultMaxBufferSize = 1073741824
defaultMaxOutstandingTuples

The number of tuples to process from a parallel region queue before blocking the caller. This property is optional and its default value is zero, meaning unlimited.

defaultMaxProcessingBatchSize

The maximum batch size in tuples to process from a parallel region queue at once. This property is optional and its default value is zero, meaning unlimited.

For example:

defaultMaxProcessingBatchSize = 0
defaultStorageMethod

The default storage method to use for all parallel region queues. Valid values are HEAP or TRANSACTIONAL_MEMORY. This property is optional and its default value is HEAP.

For example:

defaultStorageMethod = "HEAP"
defaultWaitStrategy

The default wait strategy used by the disruptor that controls parallel region queues. Valid values are BLOCKING, BUSY_SPIN, SLEEPING, and YIELDING. This property is optional and its default value is BLOCKING.

For example:

defaultWaitStrategy = "BLOCKING"
drainBeforeShutdown

Drain all parallel region queues before shutdown. This property is optional and its default value is true.

For example:

drainBeforeShutdown = true
expansionDelayMilliseconds

The time in milliseconds to wait for a parallel region queue to have space, before expanding it. This property is optional and its default value is 1000.

For example

expansionDelayMilliseconds = 1000
expansionDelayReportThresholdMilliseconds

When a parallel region queue expansion has to wait, report the wait time if above this threshold in milliseconds. A value of zero means do not report. This property is optional and its default value is zero.

For example:

expansionDelayReportThresholdMilliseconds = 1000
perQueueHeapMemoryThresholdPercent

The percentage of total memory to allow a parallel region queue to take up, before switching to blocking and linear growth. This value only applies to queues that use heap memory for storage. This property is optional and its default value is one percent.

For example:

perQueueHeapMemoryThresholdPercent = 1
engineMonitor

Configures the engine's monitoring subsystem. This object is optional.

enabled

Bool. This is required in order to use the engine monitor. This property is optional and if it is not present, defaults to false. If this property is false then all all other properties are ignored.

For example:

enabled = false
statsFrequencyMilliseconds

Int. Configures how frequently monitoring stats are produced. This property is optional and its default value is 1000.

For example:

statsFrequencyMilliseconds = 2000
decisionTableStatistics

Configures whether or not decision table statistics are enabled and optionally configures filters to control which decision table statistics are published. This field is optional. If this field is not specified then decision table statistics are enabled.

enabled

Bool. Whether or not decision table statistics are enabled. This property is required.

filters

Array. Defines filters to limit which decision table statistics are published. This field is optional. This field is ignored if the enabled field is set to false. If no filters are specified then statistics are published for all decision table operators. If multiple match a decision table operator name then the first filter that matches is used. If filters are specified and no filters match a decision table operator name then statistics are not published for that operator.

garbageCollectionStatistics

Configures whether or not garbage collection statistics are enabled. If this field is not specified then garbage collection statistics are enabled.

enabled

Bool. Whether or not decision table statistics are enabled. This property is required.

operatorStatistics

Configures whether or not operator statistics are enabled and optionally configures filters to control which operator statistics are published. This field is optional. If this field is not specified then operator statistics are enabled.

enabled

Bool. Whether or not decision table statistics are enabled. This property is required.

filters

Array. Defines filters to limit which decision table statistics are published. This field is optional. This field is ignored if the enabled field is set to false. If no filters are specified then statistics are published for all decision table operators. If multiple match a decision table operator name then the first filter that matches is used. If filters are specified and no filters match a decision table operator name then statistics are not published for that operator.

queueStatistics

Configures whether or not queue statistics are enabled and optionally configures filters to control which queue statistics are published. This field is optional. If this field is not specified then queue statistics are enabled.

enabled

Bool. Whether or not decision table statistics are enabled. This property is required.

enableLatencyStatistics

Bool. Whether or not to publish queue latency statistics. This field is optional and its default value is false. This field is ignored if the enabled field is set to false.

filters

Array. Defines filters to limit which decision table statistics are published. This field is optional. This field is ignored if the enabled field is set to false. If no filters are specified then statistics are published for all decision table operators. If multiple match a decision table operator name then the first filter that matches is used. If filters are specified and no filters match a decision table operator name then statistics are not published for that operator.

streamStatistics

Configures whether or not stream statistics are enabled and optionally configures filters to control which stream statistics are published. This field is optional. If this field is not specified then stream statistics are enabled.

enabled

Bool. Whether or not decision table statistics are enabled. This property is required.

filters

Array. Defines filters to limit which decision table statistics are published. This field is optional. This field is ignored if the enabled field is set to false. If no filters are specified then statistics are published for all decision table operators. If multiple match a decision table operator name then the first filter that matches is used. If filters are specified and no filters match a decision table operator name then statistics are not published for that operator.

systemStatistics

Configures whether or not system statistics are enabled. If this field is not specified then system statistics are enabled.

enabled

Bool. Whether or not decision table statistics are enabled. This property is required.

threadStatistics

Configures whether or not thread statistics are enabled. If this field is not specified then thread statistics are enabled.

enabled

Bool. Whether or not decision table statistics are enabled. This property is required.

statisticFilter

Defines a filter for enabling or disabling the publishing of statistics based upon the name of the entity (e.g. operator name, queue name, etc) and/or threshold values.

pattern

String. Regular expression pattern to specify the names of the entities that this filter applies to. This field is required.

enabled

Bool. Whether or not to publish statistics for the entities matching the pattern field. This field is optional and its default value is true.

thresholds

Array. Defines thresholds so that a statistic tuple is published only if the specified statistic value in the statistic tuple is greater than the threshold. This field is optional. If no thresholds are specified then system-defined thresholds are used. Typically this means that only non-zero values are reported but this may vary per type of statistic.

statistic

Enum. The statistic to set a threshold for. This field is required.

The statistics specified must correspond to the type of statistic that the thresholds are being defined for. For example, if defining thresholds for operator statistics then only statistics beginning with the OPERATOR_ prefix can be specified.

OPERATOR_INPUT_PORT_TUPLES_COUNT

The number of tuples on the input port of this operator in the most recent snapshot interval.

OPERATOR_OUTPUT_PORT_TUPLES_COUNT

The number of tuples on the output port of this operator in the most recent snapshot interval.

OPERATOR_PERCENT_CPU

The cumulative percentage of CPU time used by the operator since the monitored engine started.

OPERATOR_TIME_MICROSECONDS

The time in microseconds used by this operator in the most recent snapshot interval.

OPERATOR_SIZE

The size of the operator, if applicable for the operator. The meaning of the operator size is operator-specific and is not applicable to all operators. If operator size is not applicable a value of 0 is returned.

QUEUE_TUPLES_MAXIMUM_COUNT

The maximum queue backlog reached since the current application was started.

QUEUE_TUPLES_COUNT

The number of tuples passing through this queue in the most recent snapshot interval.

QUEUE_TOTAL_TIME_MICROSECONDS

Total time tuples spent queued, in microseconds. Queue latency statistics must be enabled in addition to queue statistics being enabled for this statistic to be available.

QUEUE_DEQUEUED_TUPLES_TOTAL_COUNT

Total number of tuples dequeued. Queue latency statistics must be enabled in addition to queue statistics being enabled for this statistic to be available.

STREAM_TUPLES_TOTAL_COUNT

The total number of tuples processed on the stream since engine start.

DECISION_TABLE_CONDITIONS_EVALUATED_TOTAL_COUNT

Total number of decision table conditions evaluated.

DECISION_TABLE_CONDITIONS_TOTAL_TIME_MICROSECONDS

Total number of microseconds spent evaluating decision table conditions.

DECISION_TABLE_ACTIONS_EXECUTED_TOTAL_COUNT

Total number of decision table actions executed.

DECISION_TABLE_ACTION_TOTAL_TIME_MICROSECONDS

Total number of microseconds spent running actions.

DECISION_TABLE_NON_MATCHING_TUPLES_TOTAL_COUNT

Total number of non-matching tuples emitted.

threshold

The threshold value. Only values greater than this value are published. This field is required.

timeService

Configures settings related to the Time Service. This property is optional and has defaults as specified below.

type

Time service type, one of CONTROLLABLE or WALLCLOCK. WALLCLOCK uses the system clock internally. CONTROLLABLE can be initialized with a target time as specified below. This property is optional and its default is WALLCLOCK.

For example:

type = "WALLCLOCK"
targetTime

Configures settings related to the Time Service. Initial time of the time service in milliseconds, if type above is CONTROLLABLE. If above is set to WALLCLOCK, the value is ignored. This property is optional and its default value is zero, which means initialize from the system clock.

For example:

targetTime = 2018-06-06 12:12:24.123+0000
errorHandler

Default handling for runtime errors is to discard the error-producing event and continue with processing.

evalError

This exception is caused by a variety of reasons, including errors that occur when evaluating expressions or when flushing disk-based query table logs, and incorrect timestamp formats. Continuing after such errors is not recommended.

For example:

evalError = "shutdown"
evalPluginError

Java plug-ins can get eval-plugin-error exceptions for a variety of reasons, including TupleExceptions, errors, and when setting parameters. Continuing after such errors is not recommended.

For example:

evalPluginError = "ignore"
orderingError

The Gather operator throws this error when an order by field value is null or the current value is less then a previous value.

For example:

orderingError = "continue"
constantsOverwrite

This property is deprecated as of TIBCO® Data Streams 10.5.0.

String. Constant values defined in a module may be overwritten with a value at engine startup time using this object. The object's values are constants and are substituted at engine startup time.

This optional property is encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.

For example:

constantsOverwrite = {
  constant1 = "value1"
  constant2 = "value2"
}
artifactWaitTimeoutMilliseconds

If an operator in an EventFlow module specifies that an artifact must exist before the operator is started, then this property configures how long to wait for the artifact from a TIBCO Artifact Management Server (AMS), before cancelling engine startup. This property is optional and its default value is 20000.

For example:

artifactWaitTimeoutMilliseconds = 20000
recompileBehavior

The Warm start-up feature allows engines running EventFlow fragments to be restarted without recompiling the fragment's EventFlow source code (like .sbapp files). Warm start-up is disabled by default and needs to be enabled in a StreamBase engine configuration by setting the recompileBehavior field to REUSE in the StreamBase section of the configuration.

For example:

recompileBehavior = "REUSE"

EventFlowDeployment Root Object

This section shows the configuration objects and properties that are valid for the EventFlowDeployment root object, and to specify properties of containers and modules. See the sample configuration in Full EventFlowDeployment Example.

EventFlowDeployment

Use the EventFlowDeployment root object to control the binding of EventFlow modules to containers at deployment time.

associatedWithEngines

If you want to restrict this object to be associated with specific engines, do so here. Each entry can be a specific engine name or a regular expression that can applies to more than one engine. This array is optional and has no default value. If not present, the configuration uses default engine association based on its location in a fragment or application archive, or in a node deploy configuration.

For example:

associatedWithEngines = [ "javaengine", "otherengine[0-9]" ]
containers

A collection of container configurations, indexed by container name. This parameter is optional and has no default value.

default, connectionlogic, dbwriter

String. Specifies the name for a StreamBase container. The three names above are arbitrary examples, except that StreamBase Studio automatically names the first container default.

moduleName

Specifies the top-level EventFlow module to reside in this container, expressed as an EventFlow module reference. This property is required.

moduleParameterSettings

An EventFlow parameter settings object contains a set of names and corresponding values for parameters in EventFlow modules, as well as inclusions of other parameter settings. Parameter settings are named and can be included by any container configuration or other parameter settings object.

includes

A list of zero or more parameter settings objects included by this parameter settings object. All parameter names and values in the included settings (and their included settings, and so on) become part of this parameter settings object. Multiple and nested inclusions are processed depth-first, and duplicate parameter values are overwritten, with the latest value taking precedence. Parameter names that also appear in the parameters section will supersede any included values. This property is optional and has no default value.

parameters

A set of one or more EventFlow parameters to use in expressions in any module or container run by this EventFlow engine. This property is optional and has no default value.

sampleParameterName0

Arbitrary parameter name-value binding example. Parameter names can identify module or a container parameters; scope is determined by where the enclosing settings object is used. Parameter names containing dots must be surrounded with double quotes. Parameter values are encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.

sampleParameterName0 = "sampleParameterValue"
containerParameterSettings

An EventFlow parameter settings object contains a set of names and corresponding values for parameters in EventFlow modules, as well as inclusions of other parameter settings. Parameter settings are named and can be included by any container configuration or other parameter settings object.

includes

A list of zero or more parameter settings objects included by this parameter settings object. All parameter names and values in the included settings (and their included settings, and so on) become part of this parameter settings object. Multiple and nested inclusions are processed depth-first, and duplicate parameter values are overwritten, with the latest value taking precedence. Parameter names that also appear in the parameters section will supersede any included values. This property is optional and has no default value.

parameters

A set of one or more EventFlow parameters to use in expressions in any module or container run by this EventFlow engine. This property is optional and has no default value.

sampleParameterName0

Arbitrary parameter name-value binding example. Parameter names can identify module or a container parameters; scope is determined by where the enclosing settings object is used. Parameter names containing dots must be surrounded with double quotes. Parameter values are encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.

sampleParameterName0 = "sampleParameterValue"
dequeue

One of {ENABLED | DISABLED | DROP_TUPLES} to specify the startup state of dequeuing for this container. If you specify DISABLED, dequeue attempts are actively refused and exceptions are thrown. If you specify DROP_TUPLES, tuples are silently dropped. This property is optional and its default value is ENABLED.

enqueue

One of {ENABLED | DISABLED | DROP_TUPLES} to specify the startup state of enqueuing for this container. If you specify DISABLED, enqueue attempts are actively refused and exceptions are thrown. If you specify DROP_TUPLES, tuples are silently dropped. This property is optional and its default value is ENABLED.

suspend

Set this value to true to specify that the module in this container should start up suspended. Set it to false to have the module start running when the container is added. This property is optional and its default value is false.

For example:

suspend = false
pluginFunctions

The pluginFunctions object is deprecated as of TIBCO® Data Streams 10.5.0. Migrate your custom function and alias definitions to use the CustomFunctionGroup root object and its customFunctions object.

extensionPoints

An associative array of extension point target IDs and their contents. This property is optional and has no default value.

sampleExtensionPoint0

Sample Extension Point name.

extensions

The ID of this Extension Point, which must match the ID specified in the StreamBase Studio Properties view for an Extension Point operator in a module that will use this configuration file.

ExtensionPointID

Specify an Extension Point ID defined in an Extension Point operator.

moduleName

An EventFlow module reference for a module that implements the interface in the enclosing extension point operator. This property is required.

For example:

moduleName = "com.example.sampleModuleName"
moduleParameterSettings

An EventFlow parameter settings object contains a set of names and corresponding values for parameters in EventFlow modules, as well as inclusions of other parameter settings. Parameter settings are named and can be included by any container configuration or other parameter settings object.

includes

A list of zero or more parameter settings objects included by this parameter settings object. All parameter names and values in the included setting(s) (and their included settings, and so on) become part of this parameter settings object. Multiple and nested inclusions are processed depth-first, and duplicate parameter values are overwritten, with the latest value taking precedence. Parameter names that also appear in the parameters section will supersede any included values. This property is optional and has no default value.

parameters

A set of one or more EventFlow parameters to use in expressions in any module or container run by this EventFlow engine. This property is optional and has no default value. Parameter names can identify module or a container parameter; scope is determined by where the enclosing settings object is used. This optional property is encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.

trace

Configures runtime tracing for this container's module. This array is optional and has no default value.

buffered

If true, the trace system buffers trace output before writing to a file. If false, the output trace file is flushed on every trace line. This property is optional and its default value is true.

For example:

buffered = false
compress

If true, the trace system compresses its files using gzip. If false, files remain uncompressed. This property is optional and its default value is false.

For example:

compress = true
fileBase

String. Absolute base file name for trace files. This optional property has no default value. If absent, trace output is written to the console.

For example:

fileBase = "/absolute/path/trace-"
matchOperatorAndStreamRegexp

A regular expression pattern that limits tracing to the operators and streams whose name matches the pattern. This parameter is optional and has no default value.

overwrite

Bool. If true, all trace files are overwritten on engine restart. If false, trace files are appended to on engine restart. This property is optional and its default value is false.

For example:

overwrite = true
containerConnections

A collection of container connections used by containers in this engine. This array is optional and has no default value. A container connection describes a means for connecting streams in different application modules running in different containers.

destination

An address describing the entity (usually an input stream) that will consume the output of the connected source container. This property is required.

For example:

destination = "default.ProcessTransactionAndScore.
  ProcessTransaction.EvaluateModel.default.Control"
source

An address describing the entity in another named container (usually an output stream) that will send the data to the destination container. This property is required.

For example:

source = "zkMaintainModel.ModelUpdates"
filter

A predicate expression filter to limit the tuples that cross to the destination container. The expression to enter must resolve to true or false, and should match against one or more properties in the schema of the connection's stream.

This property is optional and has no default value.

For example:

filter = "this is a filter"
synchronicity

By default, connections between containers are set up to run asynchronously, with a queue to manage the flow of tuples through the connection. StreamBase also supports synchronous container connections, which are low latency, direct connections between containers, and do not have queue management. This property is optional and its default value is ASYNCHRONOUS.

For example:

synchronicity = "ASYNCHRONOUS"

ParameterSettingsGroup Root Object

This section shows the configuration objects and properties that are valid for the ParameterSettingsGroup root object. See the sample configuration in Full ParameterSettingsGroup Example.

ParameterSettingsGroup

An EventFlow parameter settings group is an associative array of parameter settings objects, indexed by name.

associatedWithEngines

Configuration types that have affinity with particular engines can use this array to associate themselves with particular engines. Each element is an engine name or a regular expression matching zero or more engine names. A null value means that the instance uses default engine association. Using this property is recommended only when manually loading configurations via the node management command line tool or API; affinity for configurations in fragment archives is automatically defaulted during node installation. This array is optional and has no default value.

For example:

associatedWithEngines = [ "sample-engine-identifier" ]
parameterSettings

An associative array of parameter settings objects, indexed by name. Parameter settings object names containing dots must be surrounded with double quotes. This array is required. Parameter settings are named and can be included by any container configuration or other parameter settings object.

sample-parameter-settings-0

An example name of an EventFlow parameter settings object containing a set of names and corresponding values for parameters in EventFlow modules, as well as inclusions of other parameter settings.

includes

A list of zero or more parameter settings objects included by this parameter settings object. All parameter names and values in the included settings (and their respective included settings, and so on) become part of this parameter settings object. Multiple and nested inclusions are processed depth-first, and duplicate parameter values are overwritten, with the latest value taking precedence. Parameter names that also appear in the parameters section supersede any included values. This array is optional and has no default value.

For example:

includes = [ "my-referenced-parameter-settings" ]
parameters

A set of one or more EventFlow parameter name-value bindings to use in expressions in any module or container run by this EventFlow engine. This object is optional and has no default value.

sampleParameterName0

Arbitrary parameter name-value binding example. Parameter names can identify module or a container parameters; scope is determined by where the enclosing settings object is used. Parameter names containing dots must be surrounded with double quotes. Parameter values are encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.

sampleParameterName0 = "sampleParameterValue"

CustomFunctionGroup Root Object

This section shows the configuration objects and properties that are valid for the CustomFunctionGroup root object. See the sample configuration in Full CustomFunctionGroup Example.

CustomFunctionGroup

An EventFlow custom function group is an associative array of custom function definitions, indexed by name.

associatedWithEngines

Configuration types that have affinity with particular engines can use this array to associate themselves with particular engines. Each element is an engine name or a regular expression matching zero or more engine names. A null value means that the instance uses default engine association. Using this property is recommended only when manually loading configurations via the node management command line tool or API; affinity for configurations in fragment archives is automatically defaulted during node installation. This array is optional and has no default value.

customFunctions

This is an associative array of custom function definitions, indexed by EventFlow-visible function name. Function names must conform to Java identifier semantics, but need not match the underlying Java method name. This object is required.

Configure the customFunctions object to register one or more custom functions with the containing EventFlow engine, and to optionally assign an alias to it, or to assign an alias to a method in any Java class on the classpath.

Note that you can use the expression language's calljava() function to call any custom function or standard library function by its fully qualified class and method name, as long as the class is accessible to the current project's Maven search path. In this case, neither registering the function nor an alias name is required. By contrast, registering a function allows you to call it by its simple method name or alias name in expression language contexts.

For example:

configuration = {
  CustomFunctionGroup = {
    customFunctions = {
      hypotenuse = {
        type = "simple"
        methodName = "calculate"
        className = "com.tibco.ep.sample.Hypotenuse"
        autoArguments = true     
      }
      hypot = {
        type = "simple"
        methodName = "hypot"
        className = "java.lang.Math"
        autoArguments = true
      }
    }
  }
}

Use the following table as a guide to help you port your function definitions from the pluginFunctions object used in previous releases to this customFunctions object used in release 10.5.0 and later. Note that the pluginFunctions syntax changed in release 10.4.3 to allow more than one alias for the same function.

object or property In 10.5.0 and later In 10.4.3 and later 10.4.x In 10.4.2 and earlier
customFunctions Indexed by EventFlow-visible function name, which can be an alias. Not used. Not used.
pluginFunctions Not used. Indexed by function name or placeholder name. Indexed by function name.
java Not used. Required first object under pluginFunctions. Required first object under pluginFunctions.
alias Not used. Required. Required.
methodName Required for simple functions.

For aggregate functions, always optional, but must be accumulate if used.

For simple functions, required when configuring more than one alias for the same method; not required otherwise.

For aggregate functions, always optional, but must be accumulate if used.

Not used. Methods are named in the pluginFunctions index entry.
function-name

Configuration for a single Java function. The function-name must conform to Java identifier semantics, but does not need to match the methodName property for this function, in which case the function-name specifies an alias for that method. For example:

fn = {
  type = "simple"
  methodName = "wizardry"
  className = "com.example.dept.Sbfuncs"
  autoArguments = "true"
}

In expression language contexts, you can call this method the long way with:

calljava("com.example.dept.Sbfuncs", "wizardry", "Paris", 42)

or the short way with:

fn("Paris", 42)
type

String, one of simple or aggregate. This property is optional and its default value is simple. For example:

type = "simple"
methodName

String. Identifies the public static method in className to be called by this EventFlow function definition. For simple functions, this property is required.

When type = "aggregate", your custom aggregate function extends the StreamBase Client API's com.streambase.sb.operator.AggregateWindow class and its accumulate method. For this reason, methodName is optional for aggregate functions, but must be "accumulate" if expressed.

...
configuration = {
  CustomFunctionGroup = {
    customFunctions = {
      // Register a custom simple function with alias:
      calchyp = {
        type = "simple"
        methodName = "calculate"
        className = "com.tibco.ep.sample.Hypotenuse"
        autoArguments = true
      }
      // Declare an alias for a standard library function:
      hypot = {
        type = "simple"
        methodName = "hypot"
        className = "java.lang.Math"
        autoArguments = true
      }
      // Register a custom aggregate function with alias:
      enth = {
        className = "com.streambase.sample.Nth"
        type = "aggregate"
        autoArguments = true
      }
    }
  }
}
className

String. The fully qualified class name where the named function is found. The class must be locatable in the Maven search path in a standard Java library, or Java code in this project, or Java code in a dependency of this project. This property is required. For example:

className = "com.example.dept.Sbfuncs"
className = "java.lang.Math"
autoArguments

Bool. Set this property to true to have the StreamBase engine auto-detect the arguments and return type of the current function. This property is optional and its default value is false. This property is ignored if you also specify argumentTypes and/or returnType. For example:

autoArguments = true
argumentTypes

An array of function arguments and their data type information. Use this array to list your function's argument data types instead of using autoArguments. (Specifying argumentTypes disables autoArguments if also expressed for the same function declaration.) This property is optional and has no default value.

type

Argument type, which must be one of the valid StreamBase data types, and must be expressed as an object. This property is required. For example:

argumentTypes = [
  { type = "double" }
  { type = "long" }
  { type = "list"
      elementType = {...}
  }
  { type = "tuple"
      schema = {...}
  }
]
elementType

Defines the data type for elements of a list when an argument type is list. This object is required when the function argument's type is list and not allowed otherwise.

schema

Defines the schema for a value when the argument type is tuple. This object is required when the function argument's type is tuple and is not allowed otherwise. For example:

{ type = "tuple"
  schema = {
    name = "s1"
    fields = {
      f1 = {type = "string"}
      f2 = {type = "int"}
    }
  }
}
name

The name of this schema. This property is required.

fields

Schema fields. At least one field definition is required.

fieldname1

Schema field definition example name.

type

This tuple field's type, which must be one of the valid StreamBase data types and must be expressed as an object. This property is required for each schema field:

fieldname1 = {type = "string"}

Note that a tuple field definition can have list fields, which requires an elementType for those fields, and can have tuple fields of their own, which requires an inner schema definition for each sub-tuple field.

The Full CustomFunctionGroup Example below shows a full use of the argumentTypes and returnType parameters for the showme function.

returnType

Function return type. (Specifying returnType disables autoArguments if also expressed for the same function declaration.) This property is optional and has no default value.

type

Argument type, must be one of the valid StreamBase data types, and must be expressed as an object. This property is required. For example:

returnType = {type = "int"}
isVarargs

Set this to true if the last argument is variable length and you want to declare the individual argument elements. In this case, do not set autoArguments = true. When you use autoArguments = true, the isVarargs attribute is ignored. The default value is false. For example:

isVarargs = true

Full StreamBaseEngine Example

The following is an example of the sbengine configuration type with StreamBaseEngine root object. This shows an example of every element in use at once, which is very unlikely in a real-world project.

name = "sbengineSB"
version = "1.0.0"
type = "com.tibco.ep.streambase.configuration.sbengine"
configuration = {
  StreamBaseEngine = {
    associatedWithEngines = [ "javaengine", "otherengine[0-9]" ]
    externalClassPath = [
      "/absolute/path/d.jar"
      "/absolute/path/g.jar"
      "/absolute/path/root" 
    ]
    jvmTuning = {
      minimumDispatchThreads = 10
    }
    systemProperties = {
      prop1 = "val1"
      prop2 = "val2"
      "prop.3" = "val3" 
    }
    jvmArgs = [
      "-Xmx3g"
      "-Xms512m"
      "-XX:+UseG1GC"
      "-XX:MaxGCPauseMillis=500"
      "-XX:ConcGCThreads=1"
    ]
    streamBase = {
      dataAreaPath = "/tmp/myDataDir"
      operatorParameters = {
        MyName = "MyValue"
        // The following shows an encrypted password broken onto two lines, 
        // but this must be one long line in a working configuration file.
        "MyName.with.dots.quoted" = "#!v0hZ40tTIC1PKW8TrYdqyW81KLo2kHJG
        /GaYZ/K+1Z/UrhrlG1nfMFfuCJY+3Us4Vvt6jyAwiRcLk6aZN3vUXg=="
      }
      forceHighResolutionTimer = true
      ignoreUnboundCaptures = true
      schemaMaxSize = 1M
      nowImplementation = "thread"
      operatorStateChangeTimeoutMilliseconds = 10002
      engineMonitor = {
        enabled = false
        statsFrequencyMilliseconds = 2000
      }
      timeService = {
        type = "CONTROLLABLE"
        targetTime = "2016-10-01 12:12:24.123+0000"
      }
      errorHandler = {
        evalError = "shutdown"
        evalPluginError = "ignore"
        orderingError = "continue"
      }
      artifactWaitTimeoutMilliseconds = 20000
      recompileBehavior = "REUSE"
    }
  }
}

The following is an example of the engineMonitor configuration type with StreamBaseEngine root object.

name = "sample-EngineMonitor"
type = "com.tibco.ep.streambase.configuration.sbengine"
version = "1.0.0"
configuration = {

    StreamBaseEngine = {

        streamBase = {

            engineMonitor = {

                enabled = true
                statsFrequencyMilliseconds = 3000

                decisionTableStatistics = {
                    enabled = true
                    filters = [
                        // Enable decision table statistics for the
                        // default.DecisionTable operator.
                        {
                           pattern = "default.DecisionTable"
                           enabled = true
                        }

                        // Enable statistics for all DecisionTable operators in
                        // the policy container and define a threshold to limit
                        // which statistics tuples are published.
                        {
                          pattern = "policy\\..*"
                          enabled = true
                          thresholds = [
                            {
                              // A Decision Table statistic tuple is
                              // published only if one or both of these
                              // statistics exceeds the threshold value.
                              statistics = [
                                "DECISION_TABLE_CONDITIONS_TOTAL_TIME_MICROSECONDS",
                                "DECISION_TABLE_ACTION_TOTAL_TIME_MICROSECONDS"
                              ]
                              threshold = 5000000
                            }
                          ]
                        }

                        // Statistics are disabled for all other decision table
                        // operators because their names do not match either of
                        // the patterns defined above.
                    ]
                }

                garbageCollectionStatistics = {
                    enabled = false
                }

                operatorStatistics = {
                    enabled = true
                    filters = [
                        // Enable statistics for all operators but define
                        // define thresholds to limit which statistics
                        // tuples are published.
                        {
                           pattern = ".*"
                           enabled = true
                           // An Operator statistics tuple is published only if
                           // one or more of these statistics exceed the
                           // threshold values.
                           thresholds = [
                               {
                                   statistics = [ "OPERATOR_INPUT_PORT_TUPLES_COUNT" ]
                                   threshold = 10
                               }
                               {
                                   statistics = [ "OPERATOR_OUTPUT_PORT_TUPLES_COUNT" ]
                                   threshold = 50
                               }
                           ]
                        }
                    ]
                }

                queueStatistics = {
                    enabled = true
                    enableLatencyStatistics = true

                    // No filters specified so statistics are reported for
                    // all queues.
                }

                streamStatistics = {
                    enabled = true
                    filters = [
                        // Disable statistics for streams in the default
                        // container.
                        {
                            pattern = "default\\..*"
                            enabled = false
                        }
                        // And define a filter to enable statistics for
                        // all other streams otherwise statistics would be
                        // disabled for all other streams since they did not
                        // match the filter above.
                        {
                            pattern = ".*"
                            enabled = true
                        }
                    ]
                }

                systemStatistics = {
                    enabled = true
                }

                // Thread statistics are not configured so they are enabled
                // by default.
                //threadStatistics = { }
            }
        }
    }
}

Full EventFlowDeployment Example

The following is an example of the c.t.ep.streambase.c.sbengine configuration type with EventFlowDeployment root object. This shows an example of every element in use at once, which is very unlikely in a real-world project.

name = "sbengineDeployment"
version = "1.0.0"
type = "com.tibco.ep.streambase.configuration.sbengine"
configuration = {
    EventFlowDeployment = {
    
    associatedWithEngines = [ "sbengine1", "otherengine[0-9]" ]
    
    containers = {

      containerB = {
        moduleName = "com.example.dept.moduleB"        
        moduleParameterSettings = {
          includes = [ "my-referenced-parameter-settings" ]
          parameters = {
            sampleParameterName0 = "sampleParameterValue"
          }
        }                
        containerParameterSettings = {
          includes = [ "my-referenced-parameter-settings" ]
          parameters = {
            sampleParameterName0 = "sampleParameterValue"
          }
        }
        dequeue = "ENABLED"
        enqueue = "ENABLED"
        suspend = false
        extensionPoints = {
          sampleExtensionPoint0 = {
            extensions = {
              sampleExtension0 = {
                moduleName = "com.example.sampleModuleName"
                moduleParameterSettings = {
                  includes = [ "my-referenced-parameter-settings" ]
                  parameters = {
                    sampleParameterName0 = "sampleParameterValue"
                  }
                }
              }
            }
          }
        }
        trace = [ 
          {
          buffered = true
          compress = false
          fileBase = "c:/a/sample/file/base"
          matchOperatorAndStreamRegexp = "default.stream[0-9]"
          overwrite = false
          } 
        ]
      }
    }    

    containerConnections = [
      {
      destination = "sample.in"
      filter = "samplefieldname = 'samplefield'"
      source = "sample.in"
      synchronicity = "ASYNCHRONOUS"
      } 
    ]    
  }
}

Full ParameterSettingsGroup Example

The following is an example of the sbengine configuration type with the ParameterSettingsGroup root object.

name = "sbengineParams"
type = "com.tibco.ep.streambase.configuration.sbengine"
version = "1.0.0"
configuration = {
  ParameterSettingsGroup = {
    associatedWithEngines = [ "sample-engine-identifier" ]
    parameterSettings = {
      paramsForProjectX = {
        includes = [ "my-referenced-parameter-settings" ]
        parameters = {
          sampleParameterName0 = "sampleParameterValue"
        }
      }
    }
  }
}

Full CustomFunctionGroup Example

The following is an example of the sbengine configuration type with the CustomFunctionGroup root object. Notice that tenuse and calchyp are different aliases for the same custom function, while enth and oneline are different aliases for the same aggregate function.

name = "sbengineFunctions"
type = "com.tibco.ep.streambase.configuration.sbengine"
version = "1.0.0"
configuration = {
  CustomFunctionGroup = {
    customFunctions = {
      // Register a custom function with no alias:
      runthis = {
        type = "simple"
        methodName = "runthis"
        className = "com.tibco.ep.sample.RunUtils"
        autoArguments = true
      }
      // Register a custom function with two aliases:
      tenuse = {
        type = "simple"
        methodName = "calculate"
        className = "com.tibco.ep.sample.Hypotenuse"
        autoArguments = true     
      }
      calchyp = {
        type = "simple"
        methodName = "calculate"
        className = "com.tibco.ep.sample.Hypotenuse"
        autoArguments = true
      }
      // Declare an alias for a standard library function:
      hypot = {
        type = "simple"
        methodName = "hypot"
        className = "java.lang.Math"
        autoArguments = true
      }
      // Declare a custom aggregate function with two aliases:
      enth = {
        autoArguments = true
        className = "com.streambase.sample.Nth"
        type = "aggregate"
      }
      oneline = {
        autoArguments = true
        className = "com.streambase.sample.Nth"
        type = "aggregate"
      }
      // Full configuration of argument and return types:
      showme = {
        type = "simple"
        methodName = "display"
        className = "com.tibco.ep.sample.Mythic"
        argumentTypes = [
          { type = "double" }
          { type = "long" }
          { type = "list"
              elementType = {type = "int"}
          }
          { type = "tuple"
              schema = { 
                name = "s1"
                fields = {
                  f1 = {type = "string"}
                  f2 = {type = "int"}
                  f3 = {type = "list"
                    elementType = {type = "long"}
                  }
                }
              }
          }
        ]
        returnType = {type = "double"}
      }
    }
  }
}

Limitations

The StreamBase engine configuration type does not allow new configurations to be activated while the engine is running. It will not be possible to update EventFlow statistics settings at runtime.

Management commands could be added to enable/disable statistics, define filters, and define thresholds so that these settings can be changed at runtime.