Contents
- Overview
- Required Header Lines
- StreamBaseEngine Root Object
- EventFlowDeployment Root Object
- ParameterSettingsGroup Root Object
- CustomFunctionGroup Root Object
- Full StreamBaseEngine Example
- Full EventFlowDeployment Example
- Full ParameterSettingsGroup Example
- Full CustomFunctionGroup Example
- Limitations
This article provides a reference for writing a StreamBase engine configuration file where the HOCON type is com.tibco.ep.streambase.configuration.sbengine
.
The root objects in this configuration type are:
- StreamBaseEngine
-
This root object contains data such as custom plug-in function definitions and module definitions. When the StreamBase engine starts, information in this configuration is combined with information from all the separate configuration types to create a consistent configuration for its StreamBase server instance to use.
- EventFlowDeployment
-
This root object can contain modules, container connections, and operator parameters that can be set independently of the rest of a StreamBase engine's configuration.
- ParameterSettingsGroup
-
An EventFlow parameter settings group is an associative array of parameter settings objects, indexed by name.
- CustomFunctionGroup
-
An EventFlow custom function group is an associative array of custom function definitions, indexed by name.
Each configuration file must contain the following header lines, typically found at the beginning of each file:
- name
-
Specifies an arbitrary, case-sensitive string to name this configuration, which must be unique among other files with the same type, if any. Configuration files can refer to each other by this name. Select a name that reminds you of this configuration's type and purpose. For example:
name = "mysbengine"
- version
-
Specifies an arbitrary version number that you can use to keep track of file versions for this configuration type in your development project. The maintenance of version numbers is under user control; StreamBase does not compare versions when loading configuration files during the fragment launch process. The version number is a string value, and can contain any combination of characters and numbers. For example:
version = "1.0.0"
- type
-
This essential setting specifies the unique HOCON configuration type described on this page.
type = "com.tibco.ep.streambase.configuration.sbengine"
The three header lines taken together constitute a unique signature for each HOCON file in a project's configurations
folder. Each project's configurations
folder can contain only one file with the same signature.
The top-level configuration
object defines the configuration envelope the same way for all HOCON file types.
- configuration
-
On a line below the header lines, enter the word
configuration
followed by an open brace. Theconfiguration
object is a sibling of thename
,version
, andtype
identifiers, and serves to define the configuration envelope around this type's objects as described on this page. The file must end with the matching close brace.configuration = { ... ... }
This section shows the configuration of the objects and properties that are valid for the StreamBaseEngine root object. StreamBaseEngine is used to configure the behavior of an individual engine that runs within a StreamBase Runtime node. See the sample configuration in Full StreamBaseEngine Example.
- StreamBaseEngine
-
This root object describes a top-level StreamBase engine configuration. The StreamBase engine configuration inherits from the base Runtime Java engine.
- associatedWithEngines
-
If you want to restrict this object to be associated with specific engines, do so here. Each entry can be a specific engine name or a regular expression that can applies to more than one engine. This array is optional and has no default value. If not present, the configuration is associated with all engines.
For example:
associatedWithEngines = [ "javaengine", "otherengine[0-9]" ]
- jvmArgs
-
String. A array of JVM arguments used for this engine instance on this operating system type. These values can be overridden by individual application fragments. This array is optional and has no default value.
Do not set the classpath here. Use the guidance in Java VM Memory Settings for assistance when setting the
-Xms
(minimum) and-Xmx
(maximum) Java heap size arguments. While it is possible to specify system properties here, it is better to use the systemProperties property described below.For example:
jvmArgs = [ "-Xmx3g" "-Xms512m" "-XX:+UseG1GC" "-XX:MaxGCPauseMillis=500" "-XX:ConcGCThreads=1" ]
- systemProperties
-
A list of Java system properties that will be set in each engine instance on this operating system type. These values can be overridden and extended by individual application fragments. This object is optional and has no default value.
System properties are represented as a HOCON object, where each property name in the object is a system property name and each property value is a system property value. HOCON objects are delimited by braces.
For example:
systemProperties = { name1 = "value1" "name.2" = "value2" }
For the second example above, note the dot, which is a path separator and therefore must be inside the quoted string.
- externalClassPath
-
String. Specifies a list of JAR files and class hierarchies to add to the application fragment's classpath. This array is intended for fragments that use common external Java libraries. Paths are required to be absolute and in Java, forward-slash format, even on Windows. This array is optional and has no default value.
The third entry in this example array is a path to a folder containing a class file hierarchy:
externalClassPath = [ "/absolute/path/d.jar" "/absolute/path/g.jar" "/absolute/path/root" ]
- jvmTuning
-
Specifies various tuning parameters to pass to the Java virtual machine that runs the StreamBase server. This object is optional and has default values as described below.
- minimumDispatchThreads
-
Minimum number of dispatch threads. This property is optional and its default value is 10.
For example:
minimumDispatchThreads = 10
- maximumDispatchThreads
-
Long. Specifies the maximum number of dispatch threads. This property is optional and its default value is 2000. For example:
maximumDispatchThreads = 2000
- shutdownTimerSeconds
-
Long. Specifies the maximum number of seconds to wait for a JVM to shut down before aborting it. This property is optional and its default value is 60. For example:
shutdownTimerSeconds = 60
- timerParallelism
-
Long. Specifies the number of timers that can be executing concurrently. This property is optional and its default value is 1. For example:
timerParallelism = 1
- timerResolutionMilliseconds
-
Long. Specifies the maximum timer resolution. This is the interval at which timers are examined. Higher resolution timers have more impact on system performance. This property is optional and its default value is 1000. For example:
timerResolutionMilliseconds = 1000
- schedulerPolicy
-
The scheduling policy for the JVM process. The valid values for the policy are SCHED_FIFO, SCHED_RR, and SCHED_OTHER. This property is optional. For example:
schedulerPolicy = "SCHED_FIFO"
- schedulerPriority
-
Long. Specifies the scheduling priority for the JVM process. The valid range for priority depends on the policy; for Linux the valid values for SCHED_FIFO and SCHED_RR are 1 to 99. This property is optional and its default value is operating system-specific. For example:
schedulerPriority = 1
- streamBase
-
StreamBase EventFlow-specific configuration. The EventFlow-specific information is contained in its own sub-configuration so it does not collide with any LiveView information (where a LiveView engine configuration extends the EventFlow one).
- dataAreaPath
-
String. A path to an engine-specific data directory, which StreamBase operators, adapters, and other services use to store file data. The path can be absolute or relative; relative paths are relative to the value of the default engine data area location.
This path is created if it does not exist. If the path cannot be created or accessed, configuration audit fails. The data area can be changed at runtime by replacing a configuration with a new path. This location is never removed even if the data area is changed by replacing configuration, or the configuration is deactivated and removed.
This property is optional and has a default value of
engine-data-area
.For example:
dataAreaPath = "engine-data-area"
The default folder name is created by default in the current node's node directory, in
. See the configuration file in the Data Directory sample for examples of settings relative to the default path in Studio, and of absolute paths.nodename
/fragments/enginename
/engine-data-area - operatorParameters
-
String. An association of global parameters to use in expressions in any module or container run by this StreamBase engine. This object is optional and has no default value. Operator parameters are legacy global variables, superseded by module parameters.
An operator parameter is specified by a name and a value. The name can be a simple name such as
ParamOne
. This will:1. Expose a parameter available to modules using ${param} syntax. (This syntax also resolves to module parameters defined directly in modules, when present.)
2. Set the given value as the parameter value for any Java operator or adapter with a parameter of the given name.
The name can also be a dotted path name to set a particular adapter or operator's parameter, such as
ContainerName.OperatorName.ParamOne
. If the operator is in a submodule you must include the submodule in the dotted pathname, such asContainerName.SubModuleName.OperatorName.ParamOne
. When a path is used the parameter value will only be set for that named operator. All other parameters with the same name on other operators remain unmodified. See String Values in Parameters in the Authoring Guide for the recommended policy on quoting string parameter values.This optional property is encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.
For example:
MyName = "MyValue" "MyName.with.dots.quoted" = "#!v0hZ40tTIC1PKW8TrYdqyW81KLo2kH JG/GaYZ/K+1Z/UrhrlG1nfMFfuCJY+3Us4Vvt6jyAwiRcLk6aZN3vUXg=="
- forceHighResolutionTimer
-
Bool. Set to true to force high resolution timers, at the expense of performance. This property is optional and its default value is false.
For example:
forceHighResolutionTimer = true
- ignoreUnboundCaptures
-
Bool. StreamBase's default behavior is to fail typechecking when an unbound capture is detected. An unbound capture occurs when an input stream of a module is implemented with a capture field and that input stream is not connected to an input port of the calling Module Reference in the parent application. This situation results in an error because there is no schema to examine or fields to capture for the unbound capture instance. When you are certain your unbound captures are benign, you can optionally suppress the resulting typecheck errors. To do this, set this key to true. This property is optional and its default value is false.
- schemaMaxSize
-
Long. The maximum size of a stream's schema; that is, the total number of bytes of all properties you expect in the tuple. While there is no theoretical upper limit, in practice there is a maximum based on the available memory on the machines that will host the processing of your application. Be cautious about creating very large schemas, because applications that move smaller amounts of data perform much better than applications that move tuples containing unnecessary fields. This property is optional and its default value is 1 megabyte.
For example:
schemaMaxSize = 1M
- nowImplementation
-
What implementation should be used for calls to
now()
. Must be eithersystem
orthread
.system
uses a call to Java's System.currentTimeMillis().thread
uses a background thread that checks the time approximately every millisecond. This option results in decreased accuracy, but may be more efficient thansystem
if you callnow()
more frequently than 1000 times per second. This property is optional and its default value issystem
.For example:
nowImplementation = "thread"
- operatorStateChangeTimeoutMilliseconds
-
Int. Each Java Operator changes state along with the engine process as a whole. The engine waits for each Operator to change state before it completes its state change. The value of this property is the amount of time the engine will wait before timing out the Operator. If an operator is timed out on a state change, the engine shuts the operator down and proceeds with the state change. This property is optional and its default value is 10000.
For example:
operatorStateChangeTimeoutMilliseconds = 10002
- parallelRegionQueues
-
Engine-wide configuration settings for parallel region queues.
- defaultDataDistributionPolicy
-
The data distribution policy to use for parallel region queues. This property is only valid if the parallel region in question uses transactional memory for storage, in which case it enables high availability of that storage. Its value must be a valid data distribution policy as declared in your application configuration. This property is optional and has no default value. If it is not set, then high availability for this region's queues is not enabled.
- defaultInitialBufferSize
-
The initial buffer size for a parallel region queue to use. Must be a power of 2. These buffers can grow, so setting this is not technically necessary. This property is optional and its default value is 16.
For example:
defaultInitialBufferSize = 16
- defaultMaxBufferSize
-
The maximum queue size in tuples for a parallel region queue. This must be a power of 2. This property is optional and its default value is 2^30.
For example:
defaultMaxBufferSize = 1073741824
- defaultMaxOutstandingTuples
-
The number of tuples to process from a parallel region queue before blocking the caller. This property is optional and its default value is zero, meaning unlimited.
- defaultMaxProcessingBatchSize
-
The maximum batch size in tuples to process from a parallel region queue at once. This property is optional and its default value is zero, meaning unlimited.
For example:
defaultMaxProcessingBatchSize = 0
- defaultStorageMethod
-
The default storage method to use for all parallel region queues. Valid values are HEAP or TRANSACTIONAL_MEMORY. This property is optional and its default value is HEAP.
For example:
defaultStorageMethod = "HEAP"
- defaultWaitStrategy
-
The default wait strategy used by the disruptor that controls parallel region queues. Valid values are BLOCKING, BUSY_SPIN, SLEEPING, and YIELDING. This property is optional and its default value is BLOCKING.
For example:
defaultWaitStrategy = "BLOCKING"
- drainBeforeShutdown
-
Drain all parallel region queues before shutdown. This property is optional and its default value is true.
For example:
drainBeforeShutdown = true
- expansionDelayMilliseconds
-
The time in milliseconds to wait for a parallel region queue to have space, before expanding it. This property is optional and its default value is 1000.
For example
expansionDelayMilliseconds = 1000
- expansionDelayReportThresholdMilliseconds
-
When a parallel region queue expansion has to wait, report the wait time if above this threshold in milliseconds. A value of zero means do not report. This property is optional and its default value is zero.
For example:
expansionDelayReportThresholdMilliseconds = 1000
- perQueueHeapMemoryThresholdPercent
-
The percentage of total memory to allow a parallel region queue to take up, before switching to blocking and linear growth. This value only applies to queues that use heap memory for storage. This property is optional and its default value is one percent.
For example:
perQueueHeapMemoryThresholdPercent = 1
- engineMonitor
-
Configures the engine's monitoring subsystem. This object is optional.
- enabled
-
Bool. This is required in order to use the engine monitor. This property is optional and if it is not present, defaults to false. If this property is false then all all other properties are ignored.
For example:
enabled = false
- statsFrequencyMilliseconds
-
Int. Configures how frequently monitoring stats are produced. This property is optional and its default value is 1000.
For example:
statsFrequencyMilliseconds = 2000
- decisionTableStatistics
-
Configures whether or not decision table statistics are enabled and optionally configures filters to control which decision table statistics are published. This field is optional. If this field is not specified then decision table statistics are enabled.
- enabled
-
Bool. Whether or not decision table statistics are enabled. This property is required.
- filters
-
Array. Defines filters to limit which decision table statistics are published. This field is optional. This field is ignored if the
enabled
field is set to false. If no filters are specified then statistics are published for all decision table operators. If multiple match a decision table operator name then the first filter that matches is used. If filters are specified and no filters match a decision table operator name then statistics are not published for that operator.
- garbageCollectionStatistics
-
Configures whether or not garbage collection statistics are enabled. If this field is not specified then garbage collection statistics are enabled.
- enabled
-
Bool. Whether or not decision table statistics are enabled. This property is required.
- operatorStatistics
-
Configures whether or not operator statistics are enabled and optionally configures filters to control which operator statistics are published. This field is optional. If this field is not specified then operator statistics are enabled.
- enabled
-
Bool. Whether or not decision table statistics are enabled. This property is required.
- filters
-
Array. Defines filters to limit which decision table statistics are published. This field is optional. This field is ignored if the
enabled
field is set to false. If no filters are specified then statistics are published for all decision table operators. If multiple match a decision table operator name then the first filter that matches is used. If filters are specified and no filters match a decision table operator name then statistics are not published for that operator.
- queueStatistics
-
Configures whether or not queue statistics are enabled and optionally configures filters to control which queue statistics are published. This field is optional. If this field is not specified then queue statistics are enabled.
- enabled
-
Bool. Whether or not decision table statistics are enabled. This property is required.
- enableLatencyStatistics
-
Bool. Whether or not to publish queue latency statistics. This field is optional and its default value is false. This field is ignored if the
enabled
field is set to false. - filters
-
Array. Defines filters to limit which decision table statistics are published. This field is optional. This field is ignored if the
enabled
field is set to false. If no filters are specified then statistics are published for all decision table operators. If multiple match a decision table operator name then the first filter that matches is used. If filters are specified and no filters match a decision table operator name then statistics are not published for that operator.
- streamStatistics
-
Configures whether or not stream statistics are enabled and optionally configures filters to control which stream statistics are published. This field is optional. If this field is not specified then stream statistics are enabled.
- enabled
-
Bool. Whether or not decision table statistics are enabled. This property is required.
- filters
-
Array. Defines filters to limit which decision table statistics are published. This field is optional. This field is ignored if the
enabled
field is set to false. If no filters are specified then statistics are published for all decision table operators. If multiple match a decision table operator name then the first filter that matches is used. If filters are specified and no filters match a decision table operator name then statistics are not published for that operator.
- systemStatistics
-
Configures whether or not system statistics are enabled. If this field is not specified then system statistics are enabled.
- enabled
-
Bool. Whether or not decision table statistics are enabled. This property is required.
- threadStatistics
-
Configures whether or not thread statistics are enabled. If this field is not specified then thread statistics are enabled.
- enabled
-
Bool. Whether or not decision table statistics are enabled. This property is required.
- statisticFilter
-
Defines a filter for enabling or disabling the publishing of statistics based upon the name of the entity (e.g. operator name, queue name, etc) and/or threshold values.
- pattern
-
String. Regular expression pattern to specify the names of the entities that this filter applies to. This field is required.
- enabled
-
Bool. Whether or not to publish statistics for the entities matching the
pattern
field. This field is optional and its default value is true. - thresholds
-
Array. Defines thresholds so that a statistic tuple is published only if the specified statistic value in the statistic tuple is greater than the threshold. This field is optional. If no thresholds are specified then system-defined thresholds are used. Typically this means that only non-zero values are reported but this may vary per type of statistic.
- statistic
-
Enum. The statistic to set a threshold for. This field is required.
The statistics specified must correspond to the type of statistic that the thresholds are being defined for. For example, if defining thresholds for operator statistics then only statistics beginning with the OPERATOR_ prefix can be specified.
- OPERATOR_INPUT_PORT_TUPLES_COUNT
-
The number of tuples on the input port of this operator in the most recent snapshot interval.
- OPERATOR_OUTPUT_PORT_TUPLES_COUNT
-
The number of tuples on the output port of this operator in the most recent snapshot interval.
- OPERATOR_PERCENT_CPU
-
The cumulative percentage of CPU time used by the operator since the monitored engine started.
- OPERATOR_TIME_MICROSECONDS
-
The time in microseconds used by this operator in the most recent snapshot interval.
- OPERATOR_SIZE
-
The size of the operator, if applicable for the operator. The meaning of the operator size is operator-specific and is not applicable to all operators. If operator size is not applicable a value of 0 is returned.
- QUEUE_TUPLES_MAXIMUM_COUNT
-
The maximum queue backlog reached since the current application was started.
- QUEUE_TUPLES_COUNT
-
The number of tuples passing through this queue in the most recent snapshot interval.
- QUEUE_TOTAL_TIME_MICROSECONDS
-
Total time tuples spent queued, in microseconds. Queue latency statistics must be enabled in addition to queue statistics being enabled for this statistic to be available.
- QUEUE_DEQUEUED_TUPLES_TOTAL_COUNT
-
Total number of tuples dequeued. Queue latency statistics must be enabled in addition to queue statistics being enabled for this statistic to be available.
- STREAM_TUPLES_TOTAL_COUNT
-
The total number of tuples processed on the stream since engine start.
- DECISION_TABLE_CONDITIONS_EVALUATED_TOTAL_COUNT
-
Total number of decision table conditions evaluated.
- DECISION_TABLE_CONDITIONS_TOTAL_TIME_MICROSECONDS
-
Total number of microseconds spent evaluating decision table conditions.
- DECISION_TABLE_ACTIONS_EXECUTED_TOTAL_COUNT
-
Total number of decision table actions executed.
- DECISION_TABLE_ACTION_TOTAL_TIME_MICROSECONDS
-
Total number of microseconds spent executing actions.
- DECISION_TABLE_NON_MATCHING_TUPLES_TOTAL_COUNT
-
Total number of non-matching tuples emitted.
- threshold
-
The threshold value. Only values greater than this value are published. This field is required.
- timeService
-
Configures settings related to the Time Service. This property is optional and has defaults as specified below.
- type
-
Time service type, one of CONTROLLABLE or WALLCLOCK. WALLCLOCK uses the system clock internally. CONTROLLABLE can be initialized with a target time as specified below. This property is optional and its default is WALLCLOCK.
For example:
type = "WALLCLOCK"
- targetTime
-
Configures settings related to the Time Service. Initial time of the time service in milliseconds, if type above is CONTROLLABLE. If above is set to WALLCLOCK, the value is ignored. This property is optional and its default value is zero, which means initialize from the system clock.
For example:
targetTime = 2018-06-06 12:12:24.123+0000
- errorHandler
-
Default handling for runtime errors is to discard the error-producing event and continue with processing.
- evalError
-
This exception is caused by a variety of reasons, including errors that occur when evaluating expressions or when flushing disk-based query table logs, and incorrect timestamp formats. Continuing after such errors is not recommended.
For example:
evalError = "shutdown"
- evalPluginError
-
Java plug-ins can get eval-plugin-error exceptions for a variety of reasons, including TupleExceptions, errors, and when setting parameters. Continuing after such errors is not recommended.
For example:
evalPluginError = "ignore"
- orderingError
-
The Gather operator throws this error when an order by field value is null or the current value is less then a previous value.
For example:
orderingError = "continue"
- constantsOverwrite
-
This property is deprecated as of TIBCO® Data Streams 10.5.0.
String. Constant values defined in a module may be overwritten with a value at engine startup time using this object. The object's values are constants and are substituted at engine startup time.
This optional property is encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.
For example:
constantsOverwrite = { constant1 = "value1" constant2 = "value2" }
- artifactWaitTimeoutMilliseconds
-
If an operator in an EventFlow module specifies that an artifact must exist before the operator is started, then this property configures how long to wait for the artifact from a TIBCO Artifact Management Server (AMS), before cancelling engine startup. This property is optional and its default value is 20000.
For example:
artifactWaitTimeoutMilliseconds = 20000
- Recompile behavior
-
The Warm start-up feature allows engines running EventFlow fragments to be restarted without recompiling the fragment's EventFlow source code (e.g-.sbapp files).
Warm start-up is disabled by default and needs to be enabled in a StreamBase engine configuration by setting the "recompile behavior" field to "Reuse" in the StreamBase section of the configuration.
This section shows the configuration objects and properties that are valid for the EventFlowDeployment root object, and to specify properties of containers and modules. See the sample configuration in Full EventFlowDeployment Example.
- EventFlowDeployment
-
Use the EventFlowDeployment root object to control the binding of EventFlow modules to containers at deployment time.
- associatedWithEngines
-
If you want to restrict this object to be associated with specific engines, do so here. Each entry can be a specific engine name or a regular expression that can applies to more than one engine. This array is optional and has no default value. If not present, the configuration uses default engine association based on its location in a fragment or application archive, or in a node deploy configuration.
For example:
associatedWithEngines = [ "javaengine", "otherengine[0-9]" ]
- containers
-
A collection of container configurations, indexed by container name. This parameter is optional and has no default value.
default
,connectionlogic
,dbwriter
-
String. Specifies the name for a StreamBase container. The three names above are arbitrary examples, except that StreamBase Studio automatically names the first container
default
.- moduleName
-
Specifies the top-level EventFlow module to reside in this container, expressed as an EventFlow module reference. This property is required.
- moduleParameterSettings
-
An EventFlow parameter settings object contains a set of names and corresponding values for parameters in EventFlow modules, as well as inclusions of other parameter settings. Parameter settings are named and can be included by any container configuration or other parameter settings object.
- includes
-
A list of zero or more parameter settings objects included by this parameter settings object. All parameter names and values in the included settings (and their included settings, and so on) become part of this parameter settings object. Multiple and nested inclusions are processed depth-first, and duplicate parameter values are overwritten, with the latest value taking precedence. Parameter names that also appear in the parameters section will supersede any included values. This property is optional and has no default value.
- parameters
-
A set of one or more EventFlow parameters to use in expressions in any module or container run by this EventFlow engine. This property is optional and has no default value.
sampleParameterName0
-
Arbitrary parameter name-value binding example. Parameter names can identify module or a container parameters; scope is determined by where the enclosing settings object is used. Parameter names containing dots must be surrounded with double quotes. Parameter values are encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.
sampleParameterName0 = "sampleParameterValue"
- containerParameterSettings
-
An EventFlow parameter settings object contains a set of names and corresponding values for parameters in EventFlow modules, as well as inclusions of other parameter settings. Parameter settings are named and can be included by any container configuration or other parameter settings object.
- includes
-
A list of zero or more parameter settings objects included by this parameter settings object. All parameter names and values in the included settings (and their included settings, and so on) become part of this parameter settings object. Multiple and nested inclusions are processed depth-first, and duplicate parameter values are overwritten, with the latest value taking precedence. Parameter names that also appear in the parameters section will supersede any included values. This property is optional and has no default value.
- parameters
-
A set of one or more EventFlow parameters to use in expressions in any module or container run by this EventFlow engine. This property is optional and has no default value.
sampleParameterName0
-
Arbitrary parameter name-value binding example. Parameter names can identify module or a container parameters; scope is determined by where the enclosing settings object is used. Parameter names containing dots must be surrounded with double quotes. Parameter values are encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.
sampleParameterName0 = "sampleParameterValue"
- dequeue
-
One of {ENABLED | DISABLED | DROP_TUPLES} to specify the startup state of dequeuing for this container. If you specify DISABLED, dequeue attempts are actively refused and exceptions are thrown. If you specify DROP_TUPLES, tuples are silently dropped. This property is optional and its default value is ENABLED.
- enqueue
-
One of {ENABLED | DISABLED | DROP_TUPLES} to specify the startup state of enqueuing for this container. If you specify DISABLED, enqueue attempts are actively refused and exceptions are thrown. If you specify DROP_TUPLES, tuples are silently dropped. This property is optional and its default value is ENABLED.
- suspend
-
Set this value to true to specify that the module in this container should start up suspended. Set it to false to have the module start running when the container is added. This property is optional and its default value is false.
For example:
suspend = false
- pluginFunctions
-
The pluginFunctions object is deprecated as of TIBCO® Data Streams 10.5.0. Migrate your custom function and alias definitions to use the CustomFunctionGroup root object and its
customFunctions
object. - extensionPoints
-
An associative array of extension point target IDs and their contents. This property is optional and has no default value.
sampleExtensionPoint0
-
Sample Extension Point name.
- extensions
-
The ID of this Extension Point, which must match the ID specified in the StreamBase Studio Properties view for an Extension Point operator in a module that will use this configuration file.
ExtensionPointID
-
Specify an Extension Point ID defined in an Extension Point operator.
- moduleName
-
An EventFlow module reference for a module that implements the interface in the enclosing extension point operator. This property is required.
For example:
moduleName = "com.example.sampleModuleName"
- moduleParameterSettings
-
An EventFlow parameter settings object contains a set of names and corresponding values for parameters in EventFlow modules, as well as inclusions of other parameter settings. Parameter settings are named and can be included by any container configuration or other parameter settings object.
- includes
-
A list of zero or more parameter settings objects included by this parameter settings object. All parameter names and values in the included setting(s) (and their included settings, and so on) become part of this parameter settings object. Multiple and nested inclusions are processed depth-first, and duplicate parameter values are overwritten, with the latest value taking precedence. Parameter names that also appear in the parameters section will supersede any included values. This property is optional and has no default value.
- parameters
-
A set of one or more EventFlow parameters to use in expressions in any module or container run by this EventFlow engine. This property is optional and has no default value. Parameter names can identify module or a container parameter; scope is determined by where the enclosing settings object is used. This optional property is encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.
- trace
-
Configures runtime tracing for this container's module. This array is optional and has no default value.
- buffered
-
If true, the trace system buffers trace output before writing to a file. If false, the output trace file is flushed on every trace line. This property is optional and its default value is true.
For example:
buffered = false
- compress
-
If true, the trace system compresses its files using gzip. If false, files remain uncompressed. This property is optional and its default value is false.
For example:
compress = true
- fileBase
-
String. Absolute base file name for trace files. This optional property has no default value. If absent, trace output is written to the console.
For example:
fileBase = "/absolute/path/trace-"
- matchOperatorAndStreamRegexp
-
A regular expression pattern that limits tracing to the operators and streams whose name matches the pattern. This parameter is optional and has no default value.
- overwrite
-
Bool. If true, all trace files are overwritten on engine restart. If false, trace files are appended to on engine restart. This property is optional and its default value is false.
For example:
overwrite = true
- containerConnections
-
A collection of container connections used by containers in this engine. This array is optional and has no default value. A container connection describes a means for connecting streams in different application modules running in different containers.
- destination
-
An address describing the entity (usually an input stream) that will consume the output of the connected source container. This property is required.
For example:
destination = "default.ProcessTransactionAndScore. ProcessTransaction.EvaluateModel.default.Control"
- source
-
An address describing the entity in another named container (usually an output stream) that will send the data to the destination container. This property is required.
For example:
source = "zkMaintainModel.ModelUpdates"
- filter
-
A predicate expression filter to limit the tuples that cross to the destination container. The expression to enter must resolve to true or false, and should match against one or more properties in the schema of the connection's stream.
This property is optional and has no default value.
For example:
filter = "this is a filter"
- synchronicity
-
By default, connections between containers are set up to run asynchronously, with a queue to manage the flow of tuples through the connection. StreamBase also supports synchronous container connections, which are low latency, direct connections between containers, and do not have queue management. This property is optional and its default value is ASYNCHRONOUS.
For example:
synchronicity = "ASYNCHRONOUS"
This section shows the configuration objects and properties that are valid for the ParameterSettingsGroup root object. See the sample configuration in Full ParameterSettingsGroup Example.
- ParameterSettingsGroup
-
An EventFlow parameter settings group is an associative array of parameter settings objects, indexed by name.
- associatedWithEngines
-
Configuration types that have affinity with particular engines can use this array to associate themselves with particular engines. Each element is an engine name or a regular expression matching zero or more engine names. A null value means that the instance uses default engine association. Using this property is recommended only when manually loading configurations via the node management command line tool or API; affinity for configurations in fragment archives is automatically defaulted during node installation. This array is optional and has no default value.
For example:
associatedWithEngines = [ "sample-engine-identifier" ]
- parameterSettings
-
An associative array of parameter settings objects, indexed by name. Parameter settings object names containing dots must be surrounded with double quotes. This array is required. Parameter settings are named and can be included by any container configuration or other parameter settings object.
sample-parameter-settings-0
-
An example name of an EventFlow parameter settings object containing a set of names and corresponding values for parameters in EventFlow modules, as well as inclusions of other parameter settings.
- includes
-
A list of zero or more parameter settings objects included by this parameter settings object. All parameter names and values in the included settings (and their respective included settings, and so on) become part of this parameter settings object. Multiple and nested inclusions are processed depth-first, and duplicate parameter values are overwritten, with the latest value taking precedence. Parameter names that also appear in the parameters section supersede any included values. This array is optional and has no default value.
For example:
includes = [ "my-referenced-parameter-settings" ]
- parameters
-
A set of one or more EventFlow parameter name-value bindings to use in expressions in any module or container run by this EventFlow engine. This object is optional and has no default value.
sampleParameterName0
-
Arbitrary parameter name-value binding example. Parameter names can identify module or a container parameters; scope is determined by where the enclosing settings object is used. Parameter names containing dots must be surrounded with double quotes. Parameter values are encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.
sampleParameterName0 = "sampleParameterValue"
This section shows the configuration objects and properties that are valid for the CustomFunctionGroup root object. See the sample configuration in Full CustomFunctionGroup Example.
- CustomFunctionGroup
-
An EventFlow custom function group is an associative array of custom function definitions, indexed by name.
- associatedWithEngines
-
Configuration types that have affinity with particular engines can use this array to associate themselves with particular engines. Each element is an engine name or a regular expression matching zero or more engine names. A null value means that the instance uses default engine association. Using this property is recommended only when manually loading configurations via the node management command line tool or API; affinity for configurations in fragment archives is automatically defaulted during node installation. This array is optional and has no default value.
- customFunctions
-
This is an associative array of custom function definitions, indexed by EventFlow-visible function name. Function names must conform to Java identifier semantics, but need not match the underlying Java method name. This object is required.
Configure the
customFunctions
object to register one or more custom functions with the containing EventFlow engine, and to optionally assign an alias to it, or to assign an alias to a method in any Java class on the classpath.Note that you can use the expression language's
calljava()
function to call any custom function or standard library function by its fully qualified class and method name, as long as the class is accessible to the current project's Maven search path. In this case, neither registering the function nor an alias name is required. By contrast, registering a function allows you to call it by its simple method name or alias name in expression language contexts.For example:
configuration = { CustomFunctionGroup = { customFunctions = { hypotenuse = { type = "simple" methodName = "calculate" className = "com.tibco.ep.sample.Hypotenuse" autoArguments = true } hypot = { type = "simple" methodName = "hypot" className = "java.lang.Math" autoArguments = true } } } }
Use the following table as a guide to help you port your function definitions from the
pluginFunctions
object used in previous releases to thiscustomFunctions
object used in release 10.5.0 and later. Note that thepluginFunctions
syntax changed in release 10.4.3 to allow more than one alias for the same function.object or property In 10.5.0 and later In 10.4.3 and later 10.4.x In 10.4.2 and earlier customFunctions Indexed by EventFlow-visible function name, which can be an alias. Not used. Not used. pluginFunctions Not used. Indexed by function name or placeholder name. Indexed by function name. java Not used. Required first object under pluginFunctions. Required first object under pluginFunctions. alias Not used. Required. Required. methodName Required for simple functions. For aggregate functions, always optional, but must be
accumulate
if used.For simple functions, required when configuring more than one alias for the same method; not required otherwise. For aggregate functions, always optional, but must be
accumulate
if used.Not used. Methods are named in the pluginFunctions index entry. function-name
-
Configuration for a single Java function. The
function-name
must conform to Java identifier semantics, but does not need to match the methodName property for this function, in which case thefunction-name
specifies an alias for that method. For example:fn = { type = "simple" methodName = "wizardry" className = "com.example.dept.Sbfuncs" autoArguments = "true" }
In expression language contexts, you can call this method the long way with:
calljava("com.example.dept.Sbfuncs", "wizardry", "Paris", 42)
or the short way with:
fn("Paris", 42)
- type
-
String, one of
simple
oraggregate
. This property is optional and its default value issimple
. For example:type = "simple"
- methodName
-
String. Identifies the public static method in className to be called by this EventFlow function definition. For
simple
functions, this property is required.When
type = "aggregate"
, your custom aggregate function extends the StreamBase Client API'scom.streambase.sb.operator.AggregateWindow
class and itsaccumulate
method. For this reason, methodName is optional for aggregate functions, but must be"accumulate"
if expressed.... configuration = { CustomFunctionGroup = { customFunctions = { // Register a custom simple function with alias: calchyp = { type = "simple" methodName = "calculate" className = "com.tibco.ep.sample.Hypotenuse" autoArguments = true } // Declare an alias for a standard library function: hypot = { type = "simple" methodName = "hypot" className = "java.lang.Math" autoArguments = true } // Register a custom aggregate function with alias: enth = { className = "com.streambase.sample.Nth" type = "aggregate" autoArguments = true } } } }
- className
-
String. The fully qualified class name where the named function is found. The class must be locatable in the Maven search path in a standard Java library, or Java code in this project, or Java code in a dependency of this project. This property is required. For example:
className = "com.example.dept.Sbfuncs" className = "java.lang.Math"
- autoArguments
-
Bool. Set this property to true to have the StreamBase engine auto-detect the arguments and return type of the current function. This property is optional and its default value is false. This property is ignored if you also specify argumentTypes and/or returnType. For example:
autoArguments = true
- argumentTypes
-
An array of function arguments and their data type information. Use this array to list your function's argument data types instead of using autoArguments. (Specifying argumentTypes disables autoArguments if also expressed for the same function declaration.) This property is optional and has no default value.
- type
-
Argument type, which must be one of the valid StreamBase data types, and must be expressed as an object. This property is required. For example:
argumentTypes = [ { type = "double" } { type = "long" } { type = "list" elementType = {...} } { type = "tuple" schema = {...} } ]
- elementType
-
Defines the data type for elements of a list when an argument type is
list
. This object is required when the function argument's type islist
and not allowed otherwise. - schema
-
Defines the schema for a value when the argument type is
tuple
. This object is required when the function argument's type istuple
and is not allowed otherwise. For example:{ type = "tuple" schema = { name = "s1" fields = { f1 = {type = "string"} f2 = {type = "int"} } } }
- name
-
The name of this schema. This property is required.
- fields
-
Schema fields. At least one field definition is required.
fieldname1
-
Schema field definition example name.
- type
-
This tuple field's type, which must be one of the valid StreamBase data types and must be expressed as an object. This property is required for each schema field:
fieldname1 = {type = "string"}
Note that a tuple field definition can have list fields, which requires an elementType for those fields, and can have tuple fields of their own, which requires an inner schema definition for each sub-tuple field.
The Full CustomFunctionGroup Example below shows a full use of the argumentTypes and returnType parameters for the
showme
function. - returnType
-
Function return type. (Specifying returnType disables autoArguments if also expressed for the same function declaration.) This property is optional and has no default value.
- type
-
Argument type, must be one of the valid StreamBase data types, and must be expressed as an object. This property is required. For example:
returnType = {type = "int"}
- isVarargs
-
Set this to true if the last argument is variable length and you want to declare the individual argument elements. In this case, do not set
autoArguments = true
. When you useautoArguments = true
, the isVarargs attribute is ignored. The default value isfalse
. For example:isVarargs = true
The following is an example of the sbengine configuration type with StreamBaseEngine root object. This shows an example of every element in use at once, which is very unlikely in a real-world project.
name = "sbengineSB" version = "1.0.0" type = "com.tibco.ep.streambase.configuration.sbengine" configuration = { StreamBaseEngine = { associatedWithEngines = [ "javaengine", "otherengine[0-9]" ] externalClassPath = [ "/absolute/path/d.jar" "/absolute/path/g.jar" "/absolute/path/root" ] jvmTuning = { minimumDispatchThreads = 10 } systemProperties = { prop1 = "val1" prop2 = "val2" "prop.3" = "val3" } jvmArgs = [ "-Xmx3g" "-Xms512m" "-XX:+UseG1GC" "-XX:MaxGCPauseMillis=500" "-XX:ConcGCThreads=1" ] streamBase = { dataAreaPath = "/tmp/myDataDir" operatorParameters = { MyName = "MyValue" // The following shows an encrypted password broken onto two lines, // but this must be one long line in a working configuration file. "MyName.with.dots.quoted" = "#!v0hZ40tTIC1PKW8TrYdqyW81KLo2kHJG /GaYZ/K+1Z/UrhrlG1nfMFfuCJY+3Us4Vvt6jyAwiRcLk6aZN3vUXg==" } forceHighResolutionTimer = true ignoreUnboundCaptures = true schemaMaxSize = 1M nowImplementation = "thread" operatorStateChangeTimeoutMilliseconds = 10002 engineMonitor = { enabled = false statsFrequencyMilliseconds = 2000 } timeService = { type = "CONTROLLABLE" targetTime = "2016-10-01 12:12:24.123+0000" } errorHandler = { evalError = "shutdown" evalPluginError = "ignore" orderingError = "continue" } artifactWaitTimeoutMilliseconds = 20000 } } }
The following is an example of the engineMonitor configuration type with StreamBaseEngine root object.
name = "sample-EngineMonitor" type = "com.tibco.ep.streambase.configuration.sbengine" version = "1.0.0" configuration = { StreamBaseEngine = { streamBase = { engineMonitor = { enabled = true statsFrequencyMilliseconds = 3000 decisionTableStatistics = { enabled = true filters = [ // Enable decision table statistics for the // default.DecisionTable operator. { pattern = "default.DecisionTable" enabled = true } // Enable statistics for all DecisionTable operators in // the policy container and define a threshold to limit // which statistics tuples are published. { pattern = "policy\\..*" enabled = true thresholds = [ { // A Decision Table statistic tuple is // published only if one or both of these // statistics exceeds the threshold value. statistics = [ "DECISION_TABLE_CONDITIONS_TOTAL_TIME_MICROSECONDS", "DECISION_TABLE_ACTION_TOTAL_TIME_MICROSECONDS" ] threshold = 5000000 } ] } // Statistics are disabled for all other decision table // operators because their names do not match either of // the patterns defined above. ] } garbageCollectionStatistics = { enabled = false } operatorStatistics = { enabled = true filters = [ // Enable statistics for all operators but define // define thresholds to limit which statistics // tuples are published. { pattern = ".*" enabled = true // An Operator statistics tuple is published only if // one or more of these statistics exceed the // threshold values. thresholds = [ { statistics = [ "OPERATOR_INPUT_PORT_TUPLES_COUNT" ] threshold = 10 } { statistics = [ "OPERATOR_OUTPUT_PORT_TUPLES_COUNT" ] threshold = 50 } ] } ] } queueStatistics = { enabled = true enableLatencyStatistics = true // No filters specified so statistics are reported for // all queues. } streamStatistics = { enabled = true filters = [ // Disable statistics for streams in the default // container. { pattern = "default\\..*" enabled = false } // And define a filter to enable statistics for // all other streams otherwise statistics would be // disabled for all other streams since they did not // match the filter above. { pattern = ".*" enabled = true } ] } systemStatistics = { enabled = true } // Thread statistics are not configured so they are enabled // by default. //threadStatistics = { } } } } }
The following is an example of the c.t.ep.streambase.c.sbengine configuration type with EventFlowDeployment root object. This shows an example of every element in use at once, which is very unlikely in a real-world project.
name = "sbengineDeployment"
version = "1.0.0"
type = "com.tibco.ep.streambase.configuration.sbengine"
configuration = {
EventFlowDeployment = {
associatedWithEngines = [ "sbengine1", "otherengine[0-9]" ]
containers = {
containerB
= {
moduleName = "com.example.dept.moduleB"
moduleParameterSettings = {
includes = [ "my-referenced-parameter-settings" ]
parameters = {
sampleParameterName0 = "sampleParameterValue"
}
}
containerParameterSettings = {
includes = [ "my-referenced-parameter-settings" ]
parameters = {
sampleParameterName0 = "sampleParameterValue"
}
}
dequeue = "ENABLED"
enqueue = "ENABLED"
suspend = false
extensionPoints = {
sampleExtensionPoint0 = {
extensions = {
sampleExtension0 = {
moduleName = "com.example.sampleModuleName"
moduleParameterSettings = {
includes = [ "my-referenced-parameter-settings" ]
parameters = {
sampleParameterName0 = "sampleParameterValue"
}
}
}
}
}
}
trace = [
{
buffered = true
compress = false
fileBase = "c:/a/sample/file/base"
matchOperatorAndStreamRegexp = "default.stream[0-9]"
overwrite = false
}
]
}
}
containerConnections = [
{
destination = "sample.in"
filter = "samplefieldname = 'samplefield'"
source = "sample.in"
synchronicity = "ASYNCHRONOUS"
}
]
}
}
The following is an example of the sbengine configuration type with the ParameterSettingsGroup root object.
name = "sbengineParams"
type = "com.tibco.ep.streambase.configuration.sbengine"
version = "1.0.0"
configuration = {
ParameterSettingsGroup = {
associatedWithEngines = [ "sample-engine-identifier" ]
parameterSettings = {
paramsForProjectX
= {
includes = [ "my-referenced-parameter-settings" ]
parameters = {
sampleParameterName0 = "sampleParameterValue"
}
}
}
}
}
The following is an example of the sbengine configuration type with the CustomFunctionGroup root object. Notice that tenuse
and calchyp
are different aliases for the same custom function, while enth
and oneline
are different aliases for the same aggregate function.
name = "sbengineFunctions" type = "com.tibco.ep.streambase.configuration.sbengine" version = "1.0.0" configuration = { CustomFunctionGroup = { customFunctions = { // Register a custom function with no alias: runthis = { type = "simple" methodName = "runthis" className = "com.tibco.ep.sample.RunUtils" autoArguments = true } // Register a custom function with two aliases: tenuse = { type = "simple" methodName = "calculate" className = "com.tibco.ep.sample.Hypotenuse" autoArguments = true } calchyp = { type = "simple" methodName = "calculate" className = "com.tibco.ep.sample.Hypotenuse" autoArguments = true } // Declare an alias for a standard library function: hypot = { type = "simple" methodName = "hypot" className = "java.lang.Math" autoArguments = true } // Declare a custom aggregate function with two aliases: enth = { autoArguments = true className = "com.streambase.sample.Nth" type = "aggregate" } oneline = { autoArguments = true className = "com.streambase.sample.Nth" type = "aggregate" } // Full configuration of argument and return types: showme = { type = "simple" methodName = "display" className = "com.tibco.ep.sample.Mythic" argumentTypes = [ { type = "double" } { type = "long" } { type = "list" elementType = {type = "int"} } { type = "tuple" schema = { name = "s1" fields = { f1 = {type = "string"} f2 = {type = "int"} f3 = {type = "list" elementType = {type = "long"} } } } } ] returnType = {type = "double"} } } } }
The StreamBase engine configuration type does not allow new configurations to be activated while the engine is running. It will not be possible to update EventFlow statistics settings at runtime.
Management commands could be added to enable/disable statistics, define filters, and define thresholds so that these settings can be changed at runtime.