Using the TERR Operator

The TERR operator is a member of the Java Operators group in the Palette view in StreamBase Studio. Select the operator from the Insert an Operator or Adapter dialog. Invoke the dialog with one of the following methods:

When the dialog is open, enter terr in the search field to narrow the list of operators. Then select the Enterprise Runtime for R operator.

In order to run correctly, the operator assumes that the machine running StreamBase Server and your application has a 64-bit version of TERR version 2.7 or later installed locally. The TERR operator has been tested and validated with TERR versions 2.7, 3.0, 3.1, and 3.2.

The TERR bin directory does not need to be in the system PATH, and no environment variables are required. The TERR operator recognizes and honors the TERR_HOME environment variable if set, and if it points to the local TERR installation directory; however, setting TERR_HOME is not required.

TIBCO customers can download TERR from http://edelivery.tibco.com, or download an evaluation copy of TERR from the TIBCO Access Point.

To connect StreamBase and its TERR operator to your local TERR installation, you must either:

The TERR Operator includes the ability to specify whether the item to watch is a directory or file. A specified file to be loaded can have any file name. If it can loaded, it is; otherwise, an error is logged.

If you specify a directory, only files of with .RData extensions (note capitalization) are monitored; other file types in the specified directory that change, are added, or are deleted are ignored.

If a change is detected, the changed files are added to the changed list for all TERR instances that are available to the operator. Files can be added multiple times, but there can be at most one copy in the list at any time. When a TERR instance is acquired by an operator, the list is checked, the files loaded, and the list cleared. If a file is modified after it has been loaded, it loads again the next time the TERR instance is acquired.

Dynamic mode is the default mode of operation. In dynamic mode, input tuples are merged with the supplied script via textual replacement. With each input tuple, the values in the tuple are merged into the script by replacing a tag with the text value of the corresponding tuple. This is done for each available tag in the script, and the script is then sent to the TERR process for execution. When the script finishes, the result is retrieved and translated into an output tuple. Tags are of the form $[tagname] where tagname is the name of the StreamBase field.

The script can be supplied as data in the operator itself, or it can be read from a resource file. The script is a standard R script with one addition: fields in the script that are to be replaced by tuple data must have the format $[name] where name is the name of an input tuple field. The data in the tuple can be any supported simple data type or a list of a supported simple data type. The supported StreamBase data types for input are: string, int, boolean, double, list, and timestamp (which is converted to a string), although any type can be used as long as it converts to a string. TERR result types supported are: string, int, boolean, double, array (list), byte, factor, and dataFrame.

Script substitution is done on a purely textual basis and it is up to R to parse the results. For example, the integer 3 and the float 3 all appear the same when inserted into a script. On input, lists are handled a little differently, with the outer brackets removed. For example, the list of integers [1,2,3] is converted to the string "1,2,3". Any substitution variables should be carefully looked at as to context. In general, if a substitution can be either a scalar value or a list, it should probably be placed in a c() construct such as c($[var]), which handles both scalar and list cases.

In addition, remember that the TERR parser will be parsing the new script, and when TERR sees a number, it defaults to treating it as a double. Because of this behavior, you must prepare your script with some defensive actions. If you do want an int to be returned, you must inform the parser; do this by using the integer() function. The same is true for any numeric type: unless you do want a double, coerce the returned value to the desired type.

Strings and TimeStamps are pasted with their surrounding quotes, "string", and so will be usable immediately as strings. Other non-numeric value, such as blob, functions, capture and tuple, will come as a string but will not have the surrounding quotes and may need them to be added before use.

Booleans are treated specially. On input, a true is converted to TRUE, false is converted to FALSE and null is converted to NA; on output, the opposite conversion is done. For numbers, null is converted to NA.

An example script along with some input tuples and results are shown next. First, a TERR script:

n <- $[names]
s <- $[vals]
b <- $[bools]
result <- data.frame(n, s, b)

The input tuple (names="one", vals=2, bools=true) results in:

n <- "one"
s <- 2
b <- TRUE
result <- data.frame(n, s, b)

Here, the type of s will be double for numeric StreamBase types, double or int. To make sure s is an integer, the script would be:

s <- integer($[vals])
s <- c(integer($[vals]))

The input tuple (names=["one","two"], vals=[2,4], bools=[true,null]) results in a TERR parse error, because by default TERR considers each comma as the start of a new field. The solution is to enclose the field variables in c() constructs in your TERR script, like so:

n <- c($[names])
s <- c($[vals])
b <- c($[bools])
result <- data.frame(n, s, b)

In this case, with the same input tuple as above, the results are:

n <- c("one", "two")
s <- c(2, 4)
b <- c(TRUE, NA)
result <- data.frame(n, s, b)

Static mode is similar to dynamic operation, the difference being in how tuples are sent to TERR. Where dynamic mode uses text substitution, in static mode, the script is not modified. Instead, the fields in the input tuple are converted directly to global TERR variables. The script is then run in that environment and the result variable retrieved and converted to the output tuple. This allows the script to be very short; a simple function call is sufficient as long as the function is defined in the initially loaded model. Having the values directly converted to TERR variables greatly increases both the speed of processing and the size of the input that can be processed for each tuple.

A change in the input schema is required to implement this. All the tuple entries that are to be read into the TERR process must be in a top level tuple named terrVars. Each element in this tuple are converted to a TERR variable. If it is a simple type (int, long, double, string, bool) or a list of a simple type, it will be automatically converted. If it is a complex or enhanced type (DataFrame, Factor, or a simple type with a names column), a tuple using a special schema must be provided.

A list of ints can be sent using the tuple (1) or (list (1, 2, 3)) or the enhanced form (tuple myInts (names = "one", "two", vals=[1,2], terrType="integer"). The supported terrType values are integer, double, logical, string, list, byte, dataFrame and factor. Note that timestamp is not available in this mode and would have to be converted to strings in StreamBase before being sent to the operator. Use the Input proposed schemas button to see the actual formats.

Once the variables have been sent to the TERR process, the script is executed and the result is retrieved.

Name: Use this field to specify or change the component's name, which must be unique in the application. The name must contain only alphabetic characters, numbers, and underscores, and no hyphens or other special characters. The first character must be alphabetic or an underscore.

Operator: A read-only field that shows the formal name of the operator.

Class: A field that shows the fully qualified class name that implements the functionality of this operator. Use this class name when loading the operator in StreamSQL programs with the APPLY JAVA statement. You can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.

Start with application: If this field is set to Yes or to a module parameter that evaluates to true, an instance of this operator starts as part of the containing StreamBase Server. If this field is set to No or to a module parameter that evaluates to false, the adapter is loaded with the server, but does not start until you send an sbadmin resume command, or until you start the component with StreamBase Manager. With this option set to No or false, the operator does not start even if the application as a whole is suspended and later resumed. The recommended setting is selected by default.

Enable Error Output Port: Select this check box to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports and Error Streams to learn about Error Ports.

Description: Optionally enter text to briefly describe the component's purpose and function. In the EventFlow canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

These options are used to setup the TERR instance.

These options are used to adjust the Java environment for the TERR instance.

Use the Schemas tab to specify the schema of the output tuple for this adapter.

For general instructions on using the Edit Schema tab, see the Properties: Edit Schema Tab section of the Defining Input Streams page.

The Import proposed schemas link can be used to import schemas for the various TERR output types.

A custom schema should use the same field names for fields as the generic schema does, because those field names are what are looked at when filling the output tuple.

Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.

The TERR operator has been enhanced to work with a multiplicity greater than 1 and parallelism enabled. If you assign multiple named instances of TERR to an single operator and set the component to run in a parallel region, then each instance of the operator will use its own TERR instance, if sufficient instances were assigned. If fewer TERR instances were assigned than there are parallel operator instances, the ones available are shared among the operator instances. If more are assigned, the excess instances are unused (but will have been created). It is therefor important to assign the correct number of TERR instances.

There are two possible schemas for the data port. The first is for when the operator is in dynamic mode, where the schema is determined by the TERR script loaded at operator start time. Any updated script loaded dynamically during run time must specify inputs that have at least the same field names and data types as the initially loaded script. There may be additional fields, which are ignored.

At run time, input tuples must have at least one field whose name and data type exactly match one of the fields specified by the TERR script. Input tuples do not need to fill all fields in the TERR script, and the field order of input tuples does not need to match the TERR script's field order.

The second schema for the the data port must have a tuple field named terrVars which contains the variables to be sent to the TERR process. Any other fields are ignored.

The command port has one required item, a string named command which currently accepts a single value, reset. This causes the TERR instance to be stopped and restarted.

The status port emits tuples that describe the processing status for each input tuple. It is only present when the Output Status Tuples property is selected. The schema of the output tuple consists of four strings:

The default schema of the result port consists of:

Because the TERR operator cannot know the data type of the TERR result in advance, the terrResult field contains a subfield for each possible TERR result data type. Only one terrResult subfield is filled in per input tuple, depending on the value assigned to the result variable by the script. The other subfields are left empty (null) for each input tuple.

The top-level schema of the terrResult field is shown in the following table:

Each terrResult subfield is a tuple of lists. The first third-level field of every terrResult subfield is named names. This is a list of one or more names of the result fields of the input script.

You can specify a custom schema that contains only the result data you know to expect. For a custom schema, the names of the fields must be the same as in the default schema described above, and the sub-schemas must also match exactly.