StreamBase Engine Configuration

Contents

Overview
Required Header Lines
HOCON Elements Explained
HOCON Configuration File Examples

Overview

This article provides a reference for writing a StreamBase engine configuration file.

The StreamBase engine configuration type is a subclass of the Java engine configuration type. It contains data such as custom plug-in function definitions and module definitions. When the StreamBase engine starts, information in this configuration is combined with information from all the separate configuration types to create a consistent configuration for its StreamBase server instance to use.

An EventFlow deployment configuration object is another sbengine configuration file type (using the same package name). It can contain modules, container connections, and operator parameters that can be set independently of the rest of an EventFlow engine's configuration.

Required Header Lines

Each configuration file must contain the following header lines, typically found at the beginning of each file:

name

Specifies an arbitrary, case-sensitive string to name this configuration, which must be unique among other files with the same type, if any. Configuration files can refer to each other by this name. Select a name that reminds you of this configuration's type and purpose. For example: 

name = "mysbengine"
version

Specifies an arbitrary version number that you can use to keep track of file versions for this configuration type in your development project. The maintenance of version numbers is under user control; StreamBase does not compare versions when loading configuration files during the fragment launch process. The version number is a string value, and can contain any combination of characters and numbers. For example: 

version = "1.0.0"
type

This essential setting specifies the unique HOCON configuration type described on this page. 

type = "com.tibco.ep.streambase.configuration.sbengine"

The three header lines taken together constitute a unique signature for each HOCON file in a project's configurations folder. Each project's configurations folder can contain only one file with the same signature.

The top-level configuration object defines the configuration envelope the same way for all HOCON file types.

configuration

On a line below the header element lines, enter the word configuration followed by an open brace. The configuration element is a sibling of the name, version, and type elements, and serves to define the configuration envelope around this type's objects as described on this page. The file must end with the matching close brace. 

configuration = {
...
...
}

HOCON Elements Explained

StreamBaseEngine as Top-Level Property

The following shows the configuration's HOCON elements, its name-value pairs, usage, and syntax examples, where applicable. StreamBaseEngine is used to configure the behavior of an individual engine within a StreamBase Runtime node, almost always at design time (for example, by an application developer).

StreamBaseEngine

This proprety describes a top-level StreamBase engine configuration, which includes Java engine configuration as well.

associatedWithEngines

If you want to restrict this object to be associated with specific engines, do so here. Each entry can be a specific engine name or a regular expression that can applies to more than one engine. This name-value pair is optional and has no default value. If not present, the configuration is associated with all engines.

For example:

associatedWithEngines = [ "javaengine", "otherengine[0-9]" ]
externalClassPath

The StreamBase engine configuration inherits from the base DTM Java engine. The following properties illustrate how you can configure the Java part of the StreamBase engine.

For example, an array including the following:

externalClassPath =
[
"/absolute/path/d.jar"
"/absolute/path/g.jar"
"/absolute/path/root" (class file hierarchy)
]
jvm

Used to specify settings and properties to pass to the Java virtual machine (JVM) that runs StreamBase Server.

minimumDispatchThreads

Minimum number of dispatch threads. This name-value pair is optional and its default value is 10.

For example:

minimumDispatchThreads = 10
systemProperties

A list of Java system properties that will be set in each engine instance on this operating system type. These values can be overridden and extended by individual application fragments. This name-value pair is optional and has no default value.

System properties are represented as a HOCON object, where each property name in the object is a system property name and each property value is a system property value. HOCON objects are delimited by braces.

For example:

systemProperties = { "name1" = "value1" "name.2" = "value2" }

For the second example above, note the dot, which is a path separator and therefore must be inside the quoted string.

jvmArgs

A list of JVM arguments used in each engine instance on this operating system type. These values can be overridden by individual application fragments. This name-value pair is optional and has no default value.

For example, an array of strings containing:

jvmArgs = [
  "-Xmx3g"
  "-Xms512m"
  "-XX:+UseG1GC"
  "-XX:MaxGCPauseMillis=500"
  "-XX:ConcGCThreads=1"
  ]
streamBase

StreamBase EventFlow-specific configuration. The LiveView engine configuration extends the EventFlow one; the EventFlow-specific information is in its own sub-configuration so it does not collide with any LiveView information.

dataAreaPath

String. A path to an engine-specific data directory, which is used by operators, adapters, and other services to store data. The path can be absolute or relative; relative paths are relative to the value of the Status.ENGINE_DATA_AREA system property.

This path is created if it does not exist. If the path cannot be created or accessed, configuration audit will fail. The data area can be changed at runtime by replacing a configuration with a new path. This location is never removed even if the data area is changed by replacing configuration, or the configuration is deactivated and removed.

This name-value pair is optional and has a default value of engine-data-area.

For example:

dataAreaPath = "engine-data-area"
operatorParameters

String. An associative array of global parameters to use in expressions in any module or container run by this StreamBase engine. This name-value pair is optional and has no default value. Operator parameters are legacy global variables, superseded by module parameters.

An operator parameter is specified by a name and a value. The name can be a simple name such as ParamOne. This will:

1. Expose a parameter available to modules using ${param} syntax. (This syntax also resolves to module parameters defined directly in modules, when present).

2. Set the given value as the parameter value for any Java operator or adapter with a parameter of the given name.

The name can also be a dotted path name to set a particular adapter or operator's parameter, such as ContainerName.OperatorName.ParamOne. If the operator is in a submodule you must include the submodule in the dotted pathname, such as ContainerName.SubModuleName.OperatorName.ParamOne. When a path is used the parameter value will only be set for that named operator. All other parameters with the same name on other operators will remain unmodified. See the user documentation page String Values in Parameters in the Authoring Guide for the recommended policy on quoting string parameter values.

You can provide an enciphered string for a operator parameter's value, as generated with the sbcipher command; you must prefix #! to the generated string.

For example:

MyName = "MyValue"
"MyName.with.dots.quoted" = "#!v0hZ40tTIC1PKW8TrYdqyW81KLo2kH
   JG/GaYZ/K+1Z/UrhrlG1nfMFfuCJY+3Us4Vvt6jyAwiRcLk6aZN3vUXg=="
forceHighResolutionTimer

Bool. Set to true to force high resolution timers, at the expense of performance. This name-value pair is optional and its default value is false.

For example:

forceHighResolutionTimer = true
ignoreUnboundCaptures

Bool. StreamBase's default behavior is to fail typechecking when an unbound capture is detected. An unbound capture occurs when an input stream of a module is implemented with a capture field and that input stream is not connected to an input port of the calling Module Reference in the parent application. This situation results in an error because there is no schema to examine or fields to capture for the unbound capture instance. When you are certain your unbound captures are benign, you can optionally suppress the resulting typecheck errors. To do this, set this key to true. This name-value pair is optional and its default value is false.

schemaMaxSize

Long. The maximum size of a stream's schema; that is, the total number of bytes of all name-value pairs you expect in the tuple. While there is no theoretical upper limit, in practice there is a maximum based on the available memory on the machines that will host the processing of your application. Be cautious about creating very large schemas, because applications that move smaller amounts of data perform much better than applications that move tuples containing unnecessary name-value pairs. This name-value pair is optional and its default value is 1 megabyte.

For example:

schemaMaxSize = 1M
nowImplementation

What implementation should be used for calls to now(). Must be either system or thread.

system uses a call to Java's System.currentTimeMillis(). thread uses a background thread that checks the time approximately every millisecond. This option results in decreased accuracy, but may be more efficient than system if you call now() more frequently than 1000 times per second. This name-value pair is optional and its default value is system.

For example:

nowImplementation = "thread"
operatorStateChangeTimeoutMilliseconds

Int. Each Java Operator changes state along with the engine process as a whole. The engine waits for each Operator to change state before it completes its state change. The value of this name-value pair is the amount of time the engine will wait before timing out the Operator. If an Operator is timed out on a state change, the engine shuts the Operator down and proceeds with the state change. This name-value pair is optional and its default value is 10000.

For example:

operatorStateChangeTimeoutMilliseconds = 10002
engineMonitor

Configures the engine's monitoring subsystem. This name-value pair is optional.

For example:

enabled = false
statsFrequencyMilliseconds

Int. Configures how frequently monitoring stats are produced This name-value pair is optional and its default value is 1000.

For example:

statsFrequencyMilliseconds = 2000
timeService

Configures settings related to the Time Service. This name-value pair is optional and has defaults as specified below.

type

Time service type, one of CONTROLLABLE or WALLCLOCK. WALLCLOCK uses the system clock internally. CONTROLLABLE can be initialized with a target time as specified below. This name-value pair is optional and its default is WALLCLOCK.

For example:

type = "WALLCLOCK"
targetTime

Configures settings related to the Time Service. Initial time of the time service in milliseconds, if type above is CONTROLLABLE. If above is set to WALLCLOCK, the value is ignored. This name-value pair is optional and its default value is zero, which means initialize from the system clock.

For example:

targetTime = 123
pluginFunctions

The plugin function name-value pair is used for two reasons:

  • Registering the type signatures of custom Java functions used in the server, which are contained in JAR files.

  • Assigning shorter aliases to Java functions so they can be called in the same way as built-in functions.

This name-value pair is optional and has no default value.

java

Associative array of Java plugin-based functions, indexed by function name. This name-value pair is optional and has no default value.

For example, the following registers a custom simple function named func1 that takes a string and int argument and returns a string: 

func1 = {
  type = "simple"
  argumentTypes = {
  argument = { type = "string" }
  argument = { type = "int" }
  }
  returnType = { type = "string" }
  }

Adding an alias attribute tells the EventFlow engine to allow you to use a short function name for a given function. For example, this would allow you to call the same function in expressions in the form f(myString, myInt): 

func1 = {
  type = "simple"
  alias = "f"
  argumentTypes = {
  argument = { type = "string" }
  argument = { type = "int" }
  }
  returnType = { type = "string" }
  }

You can also specify the autoArguments key to be true and leave out the arguments and return keys to ask the EventFlow engine to auto-detect them. For example: 

log10 = {
  className = "java.lang.Math"
  alias = "log_base_ten"
  autoArguments = true
  }
type

One of simple or aggregate. This name-value pair is optional and its default value is simple.

For example:

type = "simple"
alias

String. Function alias, allowing the function to be called by the alias as well as its name. This name-value pair is required.

For example:

alias = "analias"
className

String. The fully qualified class name where the named public static function is found. This name-value pair is required.

For example:

className = "com.tibco.ep.a.b.c"
isVarargs

For Java functions only, set this to true if the last argument is variable length and you want to declare the individual argument elements. In this case, do not set autoArguments = true. When you use autoArguments = true, the isVarargs attribute is ignored.

For example:

isVarargs = false
autoArguments

Bool. For Java functions only, set this name-value pair to true to have the StreamBase engine auto-detect the arguments and return type of the current function. This name-value pair is optional and its default value is false.

For example:

autoArguments = false
argumentTypes

Function arguments. This name-value pair is optional and has no default value.

type

Argument type, must be one of the valid StreamBase data types. This name-value pair is required.

For example:

type = "tuple"
schema

Defines the schema for a value when the type is tuple. This name-value pair is required when the type tuple and not allowed otherwise.

name

The name of this complex type. This name-value pair is required.

For example:

name = "s1"
fields

Schema fields. At least one field definition is required.

f1

Schema field definition

type

Argument type, must be one of the valid StreamBase data types. This name-value pair is required.

For example:

type = "string"
schema

Defines the schema for a value when type="tuple". If present, at least one field definition is required.

For example:

schema = {
name = "s2"
fields = {
f2 = {
type = "string"}
}
elementType

Defines the data type for the elements of a list when an argument type is list. This name-value pair is required when the type is list and not allowed otherwise.

returnType

Function return type. This name-value pair is optional and has no default value.

type

Argument type, must be one of the valid StreamBase data types. This name-value pair is required.

For example:

type = "int"
errorHandler

Default handling for runtime errors is to discard the error-producing event and continue with processing.

evalError

This exception is caused by a variety of reasons, including errors that occur when evaluating expressions or when flushing disk-based query table logs, and incorrect timestamp formats. Continuing after such errors is not recommended.

For example:

evalError = "shutdown"
evalPluginError

Java plug-ins can get eval-plugin-error exceptions for a variety of reasons, including TupleExceptions, errors, and when setting parameters. Continuing after such errors is not recommended.

For example:

evalPluginError = "ignore"
orderingError

The Gather operator throws this error when an order by name-value pair value is null. The Aggregate operator throws it when an order by name-value pair value is null or the current value is less then a previous value.

For example:

orderingError = "continue"
constantsOverwrite

String. Constant values defined in a module may be overwritten with a value at engine startup time using this associative array. The array name-value pairs are constants, and the values will be substituted at engine startup time. You can provide an enciphered string for a constant's value, as generated with the sbcipher command; you must prefix #! to the generated string. This name-value pair is optional and has no default value.

For example:

constant1 = "value1"
constant2 = "value2"
artifactWaitTimeoutMilliseconds

If an operator in an EventFlow application specifies that an artifact must exist before the operator is started, then this field configures how long to wait for the artifact from a TIBCO Artifact Management Server (AMS), before aborting engine startup. This name-value pair is optional and its default value is 20000.

For example:

artifactWaitTimeoutMilliseconds = 20000

EventFlowDeployment as Top-Level Property

The following shows the configuration's HOCON elements, its name-value pairs, usage, and syntax examples, where applicable.

EventFlowDeployment

A class describing a top-level StreamBase engine configuration, which includes Java engine configuration as well.

EventFlowDeployment is used to control the binding of EventFlows to containers at deploy time. It is only meaningful in cases where there are multiple containers in a single engine, each with their own top-level “main” EventFlow.

associatedWithEngines

If you want to restrict this object to be associated with specific engines, do so here. Each entry can be a specific engine name or a regular expression that can applies to more than one engine. This name-value pair is optional and has no default value. If not present, the configuration uses default engine association based on its location in a fragment or application archive, or in a node deploy configuration.

For example:

associatedWithEngines = [ "javaengine", "otherengine[0-9]" ]
modules

List of modules to load into named containers. This name-value pair is optional. If no modules are specified below, the server creates a single container named default with no module in it.

trace

Configures runtime tracing for a StreamBase module. This is an example of data that could also be specified at deploy time via the override engine configuration.

matchOperatorAndStreamRegexp

String. A regular expression pattern that limits tracing to the operators and streams whose name matches the pattern. This name-value pair is optional and has no default value.

For example:

matchOperatorAndStreamRegexp = "stream[0-9]"
fileBase

String. Absolute base file name for trace files. This name-value pair is optional and has no default value. If this name-value pair is absent, trace output is written to the console.

For example:

fileBase = "/absolute/path/trace-"
overwrite

Bool. If true, all trace files are overwritten on engine restart. If false, trace files are appended to on engine restart. This name-value pair is optional and its default value is false.

For example:

overwrite = true
compress

Bool. If true, the trace system compresses its files using gzip. If false, files remain uncompressed. This name-value pair is optional and its default value is false.

For example:

compress = true
buffered

Bool. If true, the trace system buffers trace output before writing to a file. If false, the output trace file is flushed on every trace line. This name-value pair is optional and its default value is true.

For example:

buffered = false
moduleParameters

Specifies the value of one or more parameters to be passed to the top-level module in the current container. You can provide an enciphered string for the value; generate the enciphered string with the sbcipher command and prefix #! to the generated string. This name-value pair is optional and has no default value.

For example:

"param1.with.dots.quoted" = "value1"
param2 = "value2"
extensionPoints

An associative array of extension point target IDs and their contents. This name-value pair is optional and has no default value.

extensions

The ID of this Extension Point, which must match the ID specified in the Properties view for an Extension Point operator in an module that will use this deployment file.

For example:

foo = {
moduleName

The file system name of a module that implements this Extension Point's interface. The module must be in the fragment archive's module folder. Specify an EventFlow (.sbapp) file. This name-value pair is required.

For example:

moduleName = "module1"

moduleParameters

Specifies any module parameters for this module in this Extension Point. You can provide an enciphered string for the value; generate the enciphered string with the sbcipher command and prefix #! to the generated string. This name-value pair is optional and has no default value.

For example:

moduleParameters = {
param11 = "value11"
param12 = "value12"
moduleName

The name of an EventFlow or precompiled module file, which must exist in the fragment archive's module folder. This name-value pair is required.

For example:

moduleName = "mymodule.sbapp"
containerName

String. The name you assign to the container to hold the specified module. Container names must follow the StreamBase Identifier Naming Rules: names must begin with an alpha character or underscore, and can contain only alpha characters, numbers, and underscores. Names cannot contain hyphens, colons, or other non-alphabetic characters. For running or debugging in StreamBase Studio, the top-level module must be in a container named default. This name-value pair is required.

Note

If you use this key to define a new container, TIBCO recommends specifying a container name other than "default". This is because Studio always runs EventFlow fragments in a container with that name, and any separate container named "default" conflicts and causes a runtime error.

For example:

containerName = "mycontainer"
dequeue

One of {ENABLED | DISABLED | DROP_TUPLES} to specify the startup state of dequeuing for this container. If you specify DISABLED, then dequeue attempts are actively refused and exceptions are thrown. If you specify DROP_TUPLES, then tuples are silently dropped. This name-value pair is optional and its default value is ENABLED.

For example:

dequeue = DISABLED
enqueue

One of {ENABLED | DISABLED | DROP_TUPLES} to specify the startup state of enqueuing for this container. If you specify DISABLED, then enqueue attempts are actively refused and exceptions are thrown. If you specify DROP_TUPLES, then tuples are silently dropped. This name-value pair is optional and its default value is ENABLED.

For example:

enqueue = DROP_TUPLES
suspend

Set this value to true to specify that the module in this container should start up suspended. Set it to false to have the module start running when the container is added. This name-value pair is optional and its default value is false.

For example:

suspend = true
containerConnections

A list of container connections used between modules. This name-value pair is optional and has no default value.

source

An address describing the entity in another named container (usually an output stream) that will send the data to the destination container. This name-value pair is required.

For example:

source = "zkMaintainModel.ModelUpdates"
destination

An address describing the entity (usually an input stream) that will consume the output of the connected source container. This name-value pair is required.

For example:

destination = "default.ProcessTransactionAndScore.
  ProcessTransaction.EvaluateModel.default.Control"
filter

A predicate expression filter to limit the tuples that cross to the destination container. The expression to enter must resolve to true or false, and should match against one or more name-value pairs in the schema of the connection's stream.

This name-value pair is optional and has no default value.

For example:

filter = "this is a filter"
synchronicity

By default, connections between containers are set up to run asynchronously, with a queue to manage the flow of tuples through the connection. StreamBase also supports synchronous container connections, which are low latency, direct connections between containers, and do not have queue management. This name-value pair is optional and its default value is ASYNCHRONOUS.

For example:

synchronicity = "ASYNCHRONOUS"

HOCON Configuration File Examples

Full StreamBaseEngine Example

The following is an example of the c.t.ep.streambase.c.sbengine configuration type with StreamBaseEngine class. This shows an example of every element in use at once, which is very unlikely in a real-world project. 

name = "myengine"
version = "1.0.0"
type = "com.tibco.ep.streambase.configuration.sbengine"
configuration = {
  StreamBaseEngine = {
    associatedWithEngines = [ "javaengine", "otherengine[0-9]" ]
    externalClassPath = [
      "/absolute/path/d.jar"
      "/absolute/path/g.jar"
      "/absolute/path/root" 
    ]
    jvm = {
      minimumDispatchThreads = 10
    }
    systemProperties = {
      prop1 = "val1"
      prop2 = "val2"
      "prop.3" = "val3" 
    }
    jvmArgs = [
      "-Xmx3g"
      "-Xms512m"
      "-XX:+UseG1GC"
      "-XX:MaxGCPauseMillis=500"
      "-XX:ConcGCThreads=1"
      "-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005"
    ]
    streamBase = {
      dataAreaPath = "my-data"
      operatorParameters = {
        MyName = "MyValue"
        "MyName.with.dots.quoted" = "#!v0hZ40tTIC1PKW8TrYdqyW81KLo2kHJG
        /GaYZ/K+1Z/UrhrlG1nfMFfuCJY+3Us4Vvt6jyAwiRcLk6aZN3vUXg=="
      }
      forceHighResolutionTimer = true
      ignoreUnboundCaptures = true
      schemaMaxSize = 1M
      nowImplementation = "thread"
      operatorStateChangeTimeoutMilliseconds = 10002
      engineMonitor = {
        enabled = false
        statsFrequencyMilliseconds = 2000
      }
      timeService = {
        type = "CONTROLLABLE"
        targetTime = "2016-10-01 12:12:24.123+0000"
      }
      pluginFunctions = {
        java = {
          func1 = {
            type = "simple"
            alias = "whatever"
            className = "com.tibco.ep.a.b.c"
            isVarargs = false
            autoArguments = false
            argumentTypes = [
              {
                type = "tuple"
                schema = {
                  name = "s1"
                  fields = {
                    f1 = {
                      type = "tuple"
                     
                      schema = {
                        name = "s2"
                        fields = {
                          f2 = {
                            type = "string"
                          }
                        }
                      }
                    }
                  }
                }
              }
              {
                type = "list"
                elementType = {
                   type = "string"
                }
              }
            ]
            returnType = {
              type = "int"
            }
          }
        }
      }
      errorHandler = {
        evalError = "shutdown"
        evalPluginError = "ignore"
        orderingError = "continue"
      }
      constantsOverwrite = {
        constant1 = "value1"
        constant2 = "value2"
      }
      artifactWaitTimeoutMilliseconds = 20000
    }
  }
}

Full EventFlowDeployment Example

The following is an example of the c.t.ep.streambase.c.sbengine configuration type with EventFlowDeployment class. This shows an example of every element in use at once, which is very unlikely in a real-world project. 

name = "mydeployment"
version = "1.0.0"
type = "com.tibco.ep.streambase.configuration.sbengine"
configuration = {
  EventFlowDeployment = {
 
    associatedWithEngines = [ "sbengine1", "otherengine[0-9]" ]
    modules = [
      {
        trace = [
          {
            matchOperatorAndStreamRegexp = "stream[0-9]"
            fileBase = "/absolute/path/trace-"
            overwrite = true
            compress = true
            buffered = false
          }
          {
            matchOperatorAndStreamRegexp = "anotherstream.+"
            fileBase = "/absolute/path/trace-another-"
            overwrite = false
            compress = true
            buffered = false
          }
        ]
        moduleParameters = {
          "param1.with.dots.quoted" = "value1"
          param2 = "value2"
        }
        extensionPoints = {
          foo = {
            extensions = {
               ext1 = {
                moduleName = "com.example.module1"
                moduleParameters = {
                  param11 = "value11"
                  param12 = "value12"
                }
              }
              ext2 = {
                moduleName = "com.example.module2"
                moduleParameters = {
                  param21 = "value21"
                  param22 = "value22"
                }
              }
            }
          }
          "bar" = {
            extensions = {
              ext3 = {
                moduleName = "com.example.module3"
                moduleParameters = {
                  param31 = "value31"
                  param32 = "value32"
                }
              }
            }
          }
        }

        moduleName = "com.example.deptname.thismodule"
        containerName = "mycontainer"
        dequeue = DISABLED
        enqueue = DROP_TUPLES
        suspend = true
      }
    ]

    containerConnections = [
      {
        source = "default.modulename.outputstreamname"
        destination = 
          "othercontainer.inputstreamname"
        filter = "where 'tradeID % 2 == 0'"
        synchronicity = "ASYNCHRONOUS"
   
      {
        source = "default.outputstreamname2"
        destination = "othercontainer2.modulename2.inputstreamname2"
      }
    ]
  }
}