Contents
This article provides a reference for writing a StreamBase engine configuration file where the HOCON type is com.tibco.ep.streambase.configuration.sbengine
.
The root objects in this configuration type are:
- StreamBaseEngine
-
This root object contains data such as custom plug-in function definitions and module definitions. When the StreamBase engine starts, information in this configuration is combined with information from all the separate configuration types to create a consistent configuration for its StreamBase server instance to use.
- EventFlowDeployment
-
This root object can contain modules, container connections, and operator parameters that can be set independently of the rest of a StreamBase engine's configuration.
Each configuration file must contain the following header lines, typically found at the beginning of each file:
- name
-
Specifies an arbitrary, case-sensitive string to name this configuration, which must be unique among other files with the same type, if any. Configuration files can refer to each other by this name. Select a name that reminds you of this configuration's type and purpose. For example:
name = "mysbengine"
- version
-
Specifies an arbitrary version number that you can use to keep track of file versions for this configuration type in your development project. The maintenance of version numbers is under user control; StreamBase does not compare versions when loading configuration files during the fragment launch process. The version number is a string value, and can contain any combination of characters and numbers. For example:
version = "1.0.0"
- type
-
This essential setting specifies the unique HOCON configuration type described on this page.
type = "com.tibco.ep.streambase.configuration.sbengine"
The three header lines taken together constitute a unique signature for each HOCON file in a project's configurations
folder. Each project's configurations
folder can contain only one file with the same signature.
The top-level configuration
object defines the configuration envelope the same way for all HOCON file types.
- configuration
-
On a line below the header lines, enter the word
configuration
followed by an open brace. Theconfiguration
object is a sibling of thename
,version
, andtype
identifiers, and serves to define the configuration envelope around this type's objects as described on this page. The file must end with the matching close brace.configuration = { ... ... }
The following shows the configuration's HOCON properties, usage, and syntax examples, where applicable. StreamBaseEngine is used to configure the behavior of an individual engine within a StreamBase Runtime node, almost always at design time (for example, by an application developer).
- StreamBaseEngine
-
This object describes a top-level StreamBase engine configuration, which includes Java engine configuration as well.
- associatedWithEngines
-
If you want to restrict this object to be associated with specific engines, do so here. Each entry can be a specific engine name or a regular expression that can applies to more than one engine. This array is optional and has no default value. If not present, the configuration is associated with all engines.
For example:
associatedWithEngines = [ "javaengine", "otherengine[0-9]" ]
- externalClassPath
-
The StreamBase engine configuration inherits from the base Runtime Java engine. The following array illustrates how you can configure the Java part of the StreamBase engine.
For example:
externalClassPath = [ "/absolute/path/d.jar" "/absolute/path/g.jar" "/absolute/path/root" (class file hierarchy) ]
- jvm
-
Used to specify properties to pass to the Java virtual machine (JVM) that runs StreamBase Server.
- minimumDispatchThreads
-
Minimum number of dispatch threads. This property is optional and its default value is 10.
For example:
minimumDispatchThreads = 10
- systemProperties
-
A list of Java system properties that will be set in each engine instance on this operating system type. These values can be overridden and extended by individual application fragments. This object is optional and has no default value.
System properties are represented as a HOCON object, where each property name in the object is a system property name and each property value is a system property value. HOCON objects are delimited by braces.
For example:
systemProperties = { "name1" = "value1" "name.2" = "value2" }
For the second example above, note the dot, which is a path separator and therefore must be inside the quoted string.
- jvmArgs
-
A array of JVM arguments used in each engine instance on this operating system type. These values can be overridden by individual application fragments. This array is optional and has no default value.
For example:
jvmArgs = [ "-Xmx3g" "-Xms512m" "-XX:+UseG1GC" "-XX:MaxGCPauseMillis=500" "-XX:ConcGCThreads=1" ]
- streamBase
-
StreamBase EventFlow-specific configuration. The EventFlow-specific information is contained in its own sub-configuration so it does not collide with any LiveView information (where a LiveView engine configuration extends the EventFlow one).
- dataAreaPath
-
String. A path to an engine-specific data directory, which StreamBase operators, adapters, and other services use to store data. The path can be absolute or relative; relative paths are relative to the value of the
Status.ENGINE_DATA_AREA
system property.This path is created if it does not exist. If the path cannot be created or accessed, configuration audit will fail. The data area can be changed at runtime by replacing a configuration with a new path. This location is never removed even if the data area is changed by replacing configuration, or the configuration is deactivated and removed.
This property is optional and has a default value of
engine-data-area
.For example:
dataAreaPath = "engine-data-area"
- operatorParameters
-
String. An association of global parameters to use in expressions in any module or container run by this StreamBase engine. This object is optional and has no default value. Operator parameters are legacy global variables, superseded by module parameters.
An operator parameter is specified by a name and a value. The name can be a simple name such as
ParamOne
. This will:1. Expose a parameter available to modules using ${param} syntax. (This syntax also resolves to module parameters defined directly in modules, when present.)
2. Set the given value as the parameter value for any Java operator or adapter with a parameter of the given name.
The name can also be a dotted path name to set a particular adapter or operator's parameter, such as
ContainerName.OperatorName.ParamOne
. If the operator is in a submodule you must include the submodule in the dotted pathname, such asContainerName.SubModuleName.OperatorName.ParamOne
. When a path is used the parameter value will only be set for that named operator. All other parameters with the same name on other operators remain unmodified. See String Values in Parameters in the Authoring Guide for the recommended policy on quoting string parameter values.This optional property is encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.
For example:
MyName = "MyValue" "MyName.with.dots.quoted" = "#!v0hZ40tTIC1PKW8TrYdqyW81KLo2kH JG/GaYZ/K+1Z/UrhrlG1nfMFfuCJY+3Us4Vvt6jyAwiRcLk6aZN3vUXg=="
- forceHighResolutionTimer
-
Bool. Set to true to force high resolution timers, at the expense of performance. This property is optional and its default value is false.
For example:
forceHighResolutionTimer = true
- ignoreUnboundCaptures
-
Bool. StreamBase's default behavior is to fail typechecking when an unbound capture is detected. An unbound capture occurs when an input stream of a module is implemented with a capture field and that input stream is not connected to an input port of the calling Module Reference in the parent application. This situation results in an error because there is no schema to examine or fields to capture for the unbound capture instance. When you are certain your unbound captures are benign, you can optionally suppress the resulting typecheck errors. To do this, set this key to true. This property is optional and its default value is false.
- schemaMaxSize
-
Long. The maximum size of a stream's schema; that is, the total number of bytes of all properties you expect in the tuple. While there is no theoretical upper limit, in practice there is a maximum based on the available memory on the machines that will host the processing of your application. Be cautious about creating very large schemas, because applications that move smaller amounts of data perform much better than applications that move tuples containing unnecessary fields. This property is optional and its default value is 1 megabyte.
For example:
schemaMaxSize = 1M
- nowImplementation
-
What implementation should be used for calls to
now()
. Must be eithersystem
orthread
.system
uses a call to Java's System.currentTimeMillis().thread
uses a background thread that checks the time approximately every millisecond. This option results in decreased accuracy, but may be more efficient thansystem
if you callnow()
more frequently than 1000 times per second. This property is optional and its default value issystem
.For example:
nowImplementation = "thread"
- operatorStateChangeTimeoutMilliseconds
-
Int. Each Java Operator changes state along with the engine process as a whole. The engine waits for each Operator to change state before it completes its state change. The value of this property is the amount of time the engine will wait before timing out the Operator. If an operator is timed out on a state change, the engine shuts the operator down and proceeds with the state change. This property is optional and its default value is 10000.
For example:
operatorStateChangeTimeoutMilliseconds = 10002
- engineMonitor
-
Configures the engine's monitoring subsystem. This property is optional.
- enabled
-
Bool. This is required in order to use the engine monitor. This property is optional and if it is not present, defaults are set according to the defaults of the contained types. Default is true.
For example:
enabled = false
- statsFrequencyMilliseconds
-
Int. Configures how frequently monitoring stats are produced This property is optional and its default value is 1000.
For example:
statsFrequencyMilliseconds = 2000
- timeService
-
Configures settings related to the Time Service. This property is optional and has defaults as specified below.
- type
-
Time service type, one of CONTROLLABLE or WALLCLOCK. WALLCLOCK uses the system clock internally. CONTROLLABLE can be initialized with a target time as specified below. This property is optional and its default is WALLCLOCK.
For example:
type = "WALLCLOCK"
- targetTime
-
Configures settings related to the Time Service. Initial time of the time service in milliseconds, if type above is CONTROLLABLE. If above is set to WALLCLOCK, the value is ignored. This property is optional and its default value is zero, which means initialize from the system clock.
For example:
targetTime = 2018-06-06 12:12:24.123+0000
- pluginFunctions
-
Configure the
pluginFunctions
object for any of the following reasons:-
Registering the type signatures of custom Java functions used in the server, which are contained in JAR files.
-
Assigning shorter aliases to Java functions so they can be called in the same way as built-in functions.
This object is optional and has no default value.
- java
-
Configuration of Java plugin-based functions, indexed by function name. This object is optional and has no default value.
For example, the following registers a custom simple function named
func1
that takes a string and int argument and returns a string:func1 = { type = "simple" argumentTypes = { argument = { type = "string" } argument = { type = "int" } } returnType = { type = "string" } }
Adding an
alias
attribute tells the EventFlow engine to allow you to use a short function name for a given function. For example, this would allow you to call the same function in expressions in the form f(myString, myInt):func1 = { type = "simple" alias = "f" argumentTypes = { argument = { type = "string" } argument = { type = "int" } } returnType = { type = "string" } }
You can also specify the autoArguments key to be
true
and leave out the arguments and return keys to ask the EventFlow engine to auto-detect them. For example:log10 = { className = "java.lang.Math" alias = "log_base_ten" autoArguments = true }
- type
-
One of
simple
oraggregate
. This property is optional and its default value issimple
.For example:
type = "simple"
- alias
-
String. Function alias, allowing the function to be called by the alias as well as its name. This property is required.
For example:
alias = "analias"
- className
-
String. The fully qualified class name where the named public static function is found. This property is required.
For example:
className = "com.tibco.ep.a.b.c"
- isVarargs
-
For Java functions only, set this to true if the last argument is variable length and you want to declare the individual argument elements. In this case, do not set
autoArguments = true
. When you useautoArguments = true
, the isVarargs attribute is ignored.For example:
isVarargs = false
- autoArguments
-
Bool. For Java functions only, set this property to true to have the StreamBase engine auto-detect the arguments and return type of the current function. This property is optional and its default value is false.
For example:
autoArguments = false
- argumentTypes
-
Function arguments. This property is optional and has no default value.
- type
-
Argument type, must be one of the valid StreamBase data types. This property is required.
For example:
type = "tuple"
- schema
-
Defines the schema for a value when the type is tuple. This property is required when the type tuple and not allowed otherwise.
- name
-
The name of this complex type. This property is required.
For example:
name = "s1"
- fields
-
Schema fields. At least one field definition is required.
- f1
-
Schema field definition
- type
-
Argument type, must be one of the valid StreamBase data types. This property is required.
For example:
type = "string"
- schema
-
Defines the schema for a value when
type="tuple"
. If present, at least one field definition is required.For example:
schema = { name = "s2" fields = { f2 = { type = "string"} }
- elementType
-
Defines the data type for the elements of a list when an argument type is list. This object is required when the type is list and not allowed otherwise.
- returnType
-
Function return type. This property is optional and has no default value.
- type
-
Argument type, must be one of the valid StreamBase data types. This property is required.
For example:
type = "int"
-
- errorHandler
-
Default handling for runtime errors is to discard the error-producing event and continue with processing.
- evalError
-
This exception is caused by a variety of reasons, including errors that occur when evaluating expressions or when flushing disk-based query table logs, and incorrect timestamp formats. Continuing after such errors is not recommended.
For example:
evalError = "shutdown"
- evalPluginError
-
Java plug-ins can get eval-plugin-error exceptions for a variety of reasons, including TupleExceptions, errors, and when setting parameters. Continuing after such errors is not recommended.
For example:
evalPluginError = "ignore"
- orderingError
-
The Gather operator throws this error when an order by field value is null or the current value is less then a previous value.
For example:
orderingError = "continue"
- constantsOverwrite
-
String. Constant values defined in a module may be overwritten with a value at engine startup time using this object. The object's values are constants and are substituted at engine startup time.
This optional property is encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.
For example:
constantsOverwrite = { constant1 = "value1" constant2 = "value2" }
- artifactWaitTimeoutMilliseconds
-
If an operator in an EventFlow application specifies that an artifact must exist before the operator is started, then this property configures how long to wait for the artifact from a TIBCO Artifact Management Server (AMS), before aborting engine startup. This property is optional and its default value is 20000.
For example:
artifactWaitTimeoutMilliseconds = 20000
The following shows the configuration's HOCON properties, usage, and syntax examples, where applicable.
- EventFlowDeployment
-
Use the EventFlowDeployment root object to control the binding of EventFlows to containers at deploy time. It is only meaningful in cases where there are multiple containers in a single engine, each with their own top-level “main” EventFlow.
- associatedWithEngines
-
If you want to restrict this object to be associated with specific engines, do so here. Each entry can be a specific engine name or a regular expression that can applies to more than one engine. This array is optional and has no default value. If not present, the configuration uses default engine association based on its location in a fragment or application archive, or in a node deploy configuration.
For example:
associatedWithEngines = [ "javaengine", "otherengine[0-9]" ]
- modules
-
Optional array of modules to load into named containers. If no modules are specified below, the server creates a single container named
default
with no module in it.- trace
-
Configures runtime tracing for a StreamBase module. This is an example of data that could also be specified at deploy time via the override engine configuration.
- matchOperatorAndStreamRegexp
-
String. A regular expression pattern that limits tracing to the operators and streams whose name matches the pattern. This array is optional and has no default value.
For example:
matchOperatorAndStreamRegexp = "stream[0-9]"
- fileBase
-
String. Absolute base file name for trace files. This optional property has no default value. If absent, trace output is written to the console.
For example:
fileBase = "/absolute/path/trace-"
- overwrite
-
Bool. If true, all trace files are overwritten on engine restart. If false, trace files are appended to on engine restart. This property is optional and its default value is false.
For example:
overwrite = true
- compress
-
Bool. If true, the trace system compresses its files using gzip. If false, files remain uncompressed. This property is optional and its default value is false.
For example:
compress = true
- buffered
-
Bool. If true, the trace system buffers trace output before writing to a file. If false, the output trace file is flushed on every trace line. This property is optional and its default value is true.
For example:
buffered = false
- moduleParameters
-
Specifies the value of one or more parameters to be passed to the top-level module in the current container. This optional property is encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.
For example:
moduleParameters = { "param1.with.dots.quoted" = "value1" param2 = "value2" }
- extensionPoints
-
An associative array of extension point target IDs and their contents. This property is optional and has no default value.
- extensions
-
The ID of this Extension Point, which must match the ID specified in the Properties view for an Extension Point operator in an module that will use this deployment file.
For example:
foo = {
- moduleName
-
The file system name of a module that implements this Extension Point's interface. The module must be in the fragment archive's module folder. Specify an EventFlow (.sbapp) file. This property is required.
For example:
moduleName = "module1"
- moduleParameters
-
Specifies any module parameters for this module in this Extension Point. This optional property is encryptable as part of the node-level secrecy system described in Encrypting Sensitive Configuration Data.
For example:
moduleParameters = { param11 = "value11" param12 = "value12" }
- moduleName
-
The name of an EventFlow or precompiled module file, which must exist in the fragment archive's module folder. This property is required.
For example:
moduleName = "mymodule.sbapp"
- containerName
-
String. The name you assign to the container to hold the specified module. Container names must follow the StreamBase Identifier Naming Rules: names must begin with an alpha character or underscore, and can contain only alpha characters, numbers, and underscores. Names cannot contain hyphens, colons, or other non-alphabetic characters. For running or debugging in StreamBase Studio, the top-level module must be in a container named
default
. This property is required.Note
If you use this property to define a new container, TIBCO recommends specifying a container name other than
default
. This is because Studio always runs EventFlow fragments in a container with that name, and any separate container nameddefault
conflicts and causes a runtime error.For example:
containerName = "mycontainer"
- dequeue
-
One of {ENABLED | DISABLED | DROP_TUPLES} to specify the startup state of dequeuing for this container. If you specify DISABLED, then dequeue attempts are actively refused and exceptions are thrown. If you specify DROP_TUPLES, then tuples are silently dropped. This property is optional and its default value is ENABLED.
For example:
dequeue = DISABLED
- enqueue
-
One of {ENABLED | DISABLED | DROP_TUPLES} to specify the startup state of enqueuing for this container. If you specify DISABLED, then enqueue attempts are actively refused and exceptions are thrown. If you specify DROP_TUPLES, then tuples are silently dropped. This property is optional and its default value is ENABLED.
For example:
enqueue = DROP_TUPLES
- suspend
-
Set this value to true to specify that the module in this container should start up suspended. Set it to false to have the module start running when the container is added. This property is optional and its default value is false.
For example:
suspend = true
- containerConnections
-
A list of container connections used between modules in this engine. This array is optional and has no default value. A container connection describes a means for connecting streams in different application modules running in different containers.
- source
-
An address describing the entity in another named container (usually an output stream) that will send the data to the destination container. This property is required.
For example:
source = "zkMaintainModel.ModelUpdates"
- destination
-
An address describing the entity (usually an input stream) that will consume the output of the connected source container. This property is required.
For example:
destination = "default.ProcessTransactionAndScore. ProcessTransaction.EvaluateModel.default.Control"
- filter
-
A predicate expression filter to limit the tuples that cross to the destination container. The expression to enter must resolve to true or false, and should match against one or more properties in the schema of the connection's stream.
This property is optional and has no default value.
For example:
filter = "this is a filter"
- synchronicity
-
By default, connections between containers are set up to run asynchronously, with a queue to manage the flow of tuples through the connection. StreamBase also supports synchronous container connections, which are low latency, direct connections between containers, and do not have queue management. This property is optional and its default value is ASYNCHRONOUS.
For example:
synchronicity = "ASYNCHRONOUS"
The following is an example of the c.t.ep.streambase.c.sbengine configuration type with StreamBaseEngine root object. This shows an example of every element in use at once, which is very unlikely in a real-world project.
name = "myengine" version = "1.0.0" type = "com.tibco.ep.streambase.configuration.sbengine" configuration = { StreamBaseEngine = { associatedWithEngines = [ "javaengine", "otherengine[0-9]" ] externalClassPath = [ "/absolute/path/d.jar" "/absolute/path/g.jar" "/absolute/path/root" ] jvm = { minimumDispatchThreads = 10 } systemProperties = { prop1 = "val1" prop2 = "val2" "prop.3" = "val3" } jvmArgs = [ "-Xmx3g" "-Xms512m" "-XX:+UseG1GC" "-XX:MaxGCPauseMillis=500" "-XX:ConcGCThreads=1" "-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005" ] streamBase = { dataAreaPath = "my-data" operatorParameters = { MyName = "MyValue" "MyName.with.dots.quoted" = "#!v0hZ40tTIC1PKW8TrYdqyW81KLo2kHJG /GaYZ/K+1Z/UrhrlG1nfMFfuCJY+3Us4Vvt6jyAwiRcLk6aZN3vUXg==" } forceHighResolutionTimer = true ignoreUnboundCaptures = true schemaMaxSize = 1M nowImplementation = "thread" operatorStateChangeTimeoutMilliseconds = 10002 engineMonitor = { enabled = false statsFrequencyMilliseconds = 2000 } timeService = { type = "CONTROLLABLE" targetTime = "2016-10-01 12:12:24.123+0000" } pluginFunctions = { java = { func1 = { type = "simple" alias = "whatever" className = "com.tibco.ep.a.b.c" isVarargs = false autoArguments = false argumentTypes = [ { type = "tuple" schema = { name = "s1" fields = { f1 = { type = "tuple" schema = { name = "s2" fields = { f2 = { type = "string" } } } } } } } { type = "list" elementType = { type = "string" } } ] returnType = { type = "int" } } } } errorHandler = { evalError = "shutdown" evalPluginError = "ignore" orderingError = "continue" } constantsOverwrite = { constant1 = "value1" constant2 = "value2" } artifactWaitTimeoutMilliseconds = 20000 } } }
The following is an example of the c.t.ep.streambase.c.sbengine configuration type with EventFlowDeployment root object. This shows an example of every element in use at once, which is very unlikely in a real-world project.
name = "mydeployment" version = "1.0.0" type = "com.tibco.ep.streambase.configuration.sbengine" configuration = { EventFlowDeployment = { associatedWithEngines = [ "sbengine1", "otherengine[0-9]" ] modules = [ { trace = [ { matchOperatorAndStreamRegexp = "stream[0-9]" fileBase = "/absolute/path/trace-" overwrite = true compress = true buffered = false } { matchOperatorAndStreamRegexp = "anotherstream.+" fileBase = "/absolute/path/trace-another-" overwrite = false compress = true buffered = false } ] moduleParameters = { "param1.with.dots.quoted" = "value1" param2 = "value2" } extensionPoints = { foo = { extensions = { ext1 = { moduleName = "com.example.module1" moduleParameters = { param11 = "value11" param12 = "value12" } } ext2 = { moduleName = "com.example.module2" moduleParameters = { param21 = "value21" param22 = "value22" } } } } "bar" = { extensions = { ext3 = { moduleName = "com.example.module3" moduleParameters = { param31 = "value31" param32 = "value32" } } } } } moduleName = "com.example.deptname.thismodule" containerName = "mycontainer" dequeue = DISABLED enqueue = DROP_TUPLES suspend = true } ] containerConnections = [ { source = "default.modulename.outputstreamname" destination = "othercontainer.inputstreamname" filter = "where 'tradeID % 2 == 0'" synchronicity = "ASYNCHRONOUS" { source = "default.outputstreamname2" destination = "othercontainer2.modulename2.inputstreamname2" } ] } }