StreamBase New and Noteworthy

The Insert an Operator or Adapter dialog (the one that opens when you drag the Global Operators, Adapters icon from the Palette view to the canvas) no longer shows the fully qualified path name of the class that implements each operator or adapter. When an item is selected, description text shows in the area below the title, above the OK button.

This change means that the formerly documented method of showing only Global Java Operators in this dialog no longer works. Instead, use the O, V keystroke combination for this purpose.

When you first open Studio with an empty workspace and no previous use, when you exit the Welcome screen and click the Go to Authoring link, the initial SB Authoring perspective shows a number of starting point links in the empty Property view. As of this release, a new link was added to help you import projects from the workspace of earlier releases.

The EventFlow Editor canvas rendering of data constructs, including Query Tables, is now a fixed size diamond, with the operator's name appearing in a caption underneath. The size of the diamond no longer expands to fit the table name.

Query Tables that load initial contents at startup of their containing module now show a red asterisk overlay:

Query Table delta streams are now enabled with an option setting on the Table Settings tab in the Properties view. The context menu option to enable a delta stream is now removed. If a module that has a delta stream configured in a previous release is loaded into 7.6+, the functionality remains as it was in the previous release, but the context menu now includes an option to upgrade the delta stream to the new delta stream format. Schema changes for the Query Table are now automatically propagated to delta streams.

In the Definitions tab of the EventFlow Editor, the context menu in the Constants, Named Schemas, and Table Schemas section has a new option, Import Schemas from XSD. This allows you to generate Named or Table schemas from existing XSD or XML definitions.

You can now use a constant, defined on the EventFlow Editor's Definitions tab, as a module name in an Extension Point operator.

The EventFlow Editor's Annotations tab has a new check box in the Module Annotations section: Constants in this module will be visible to any module referenced inside it. The default state of this check box is cleared. In this state, constants defined on the Definitions tab have scope only for this module, and can override only parameters defined in this module. With the check box selected, constants defined in an outer module can override parameters defined in inner modules.

The FIX Schema Designer and Thomson Reuters Schema Designer views are no longer open in the default SB Authoring perspective. Open these views as needed with Window → Show View. If you need these views open every time, consider creating a customized perspective.

The Metronome operator was updated to be able to accept the name of a dynamic variable defined in the same module as the value for the Tuple output interval field in the operator's properties. This allows you to control the emission interval of the operator at runtime. If you define the input field of the variable's associated Input Stream as a double, you can control the tick interval in fractions of a second. When you send in a new tick interval through the dynamic variable, the interval currently underway must complete before the new interval takes over. Thus, a new interval setting of 5 received at the 3rd second of a current 10-second interval must wait 7 seconds before the 5 interval takes over.

Note that if you enter an invalid interval number at runtime, such as -1, tuples are still emitted at the previously set interval, but they now contain the text of the exception from the operator. Best practice is to validate interval-setting entries at runtime to make sure they are valid values and within practical limits.

The Aggregate operator's Add or Edit Dimension dialog, when the dimension type is Field, now includes subfields of a input tuple's nested schema in the Field drop-down list.

The Properties view for Module References now has a new Constants tab. Use this tab to specify constants that can be used to override module parameters.

StreamBase's default behavior is to fail typechecking when an unbounded capture is detected. An unbounded capture occurs when the input stream of a module implementation is not connected to an input port of the module in the parent module. Compilation results in an error since there is no schema to examine or fields to capture.

This release allows you to suppress these conditions as typecheck errors temporarily during development. To do this, use a new option in the Server configuration file syntax: <global><ignore-unbound-captures/></global>.

See the discussion of the TimeService API changes below for news of the new <time-service-configurations> element.

The function data type was extended to support the following new features described in Function Data Type:

The parsejson() and parsejson_loose() functions now have an optional third argument used to specify a timestamp format with which to parse the JSON string, using format patterns in the manner of java.text.SimpleDateFormat. In the absence of a third argument, timestamps are parsed using the default format string yyyy-MM-dd HH:mm:ss.SSSZ.

Similarly, the tojson() function now has an optional third argument to specify a timestamp format with which to parse timestamp fields in the incoming tuple. When specifying the third argument, the verbosity second argument is no longer optional, and must be explicitly set.

The sbc command now accepts the name of a named schema in the --where clause of a dequeue command.

The sbc and jsbc commands were updated to emit the same results given inputs of NaN, Ininity, and -Inifity to an input stream field of type double. These values are rejected from fields of type int and long.

The filter options (-f, -Q, -t, -Y, -C, -S, -I, and -O) were removed from the sbprofile command.

Two new jsbadmin commands, getTargetTime and fastForwardTime, can retrieve the current time and, in a controlled time environment, advance the current time by a specified number of milliseconds. The new command are described on the jsbadmin reference page; see also the new Using the StreamBase JUnit TimeService page in the Test/Debug Guide.

The sbadmin addContainer command has a new option, --resourceSearch path1,path2,... Use this option to designate a resource search path for the newly added container.

The com.streambase.sb.TimeService interface in the Java Client API was updated and extended to work correctly with all time sensitive operators. This interface allows you to construct StreamBase JUnit tests for EventFlow modules that include time sensitive operators, such as a Metronome, or an Aggregate operator with a time-based window. You can also now use a feed simulation to adjust time values in a running test.

For an overview of the TimeService API, see Using the StreamBase JUnit TimeService page in the Test/Debug Guide.

The updated TimeService support provides a new way to specify whether the time values in a JUnit test run of StreamBase Server are controllable, using a new element added to the syntax of StreamBase Server Configuration files, <time-service-configurations>. For example, add:

<time-service-configurations>
  <type>CONTROLLABLE</type>
</time-service-configurations>

The updated TimeService support is accompanied by the following related changes:

Developers of custom operators or adapters can add description text using the new method setDisplayDescription() in the base Operator class. Description text appears above the OK button in the Load StreamBase Samples dialog.

The Java Client API has the following new features for developers of custom operators or adapters:

StreamBaseClient and StreamBaseAdminClient now implement the Java 8 Autocloseable feature.

The adapter collection now includes WebSocket Client Read and WebSocket Client Write adapters, with a sample in the Samples Guide.

The input ports of certain file-reading adapters is by nature asynchronous, in that file reads are buffered, and one tuple read on the input port does not result in an output tuple every time. To identify such adapters, the input ports of adapters in this category now have a circle around the port's dot and a green-colored background in the circle. Adapters in this category include: Binary File Reader, CSV Reader, External Process Operator, File Reader, FTP Operator, Regex Reader, RSS Reader, and the StreamBase Admin Operator.

The File Reader Input adapter has a new Read Files Synchronously option, for cases when you need files read in the same context instead of in the default queue method.

The Log Output adapter was updated to optionally emit its output in JSON format.

The JSON to Tuple adapter, introduced in release 7.5.2, was updated to handle incoming JSON documents with nested fields to an arbitrary depth.

The family of HDFS adapters were updated to provide a Buffer Size option for HDFS Read operator, and to provide Buffer Size, Replication Count, and Block Size options to the HDFS Write operator.

On all platforms, the 64-bit Oracle JDK version bundled with StreamBase for private use by StreamBase and Live Datamart applications is now 1.8.0_u51. JDK 8 is now required to run StreamBase Server and StreamBase Studio.

The embedded JDK is used to launch StreamBase Server, StreamBase Studio, LiveView Server, and all StreamBase and Live Datamart applications. As before, you can override this installed default to use a different supported JDK, as described in Using an External JDK.

The Java Client Libraries for StreamBase and LiveView are fully compatible with Java 7, but we recommend using Oracle Java 8, preferably 1.8.0_u51 or later, to compile and run your own client applications, custom function, custom adapters, and monitoring applications. The Java Client Libraries themselves are compiled with target set to Java 1.7 to maintain compatibility.

As before, TIBCO supports and recommends a customer-installed instance of Azul Zing for deployments of TIBCO LiveView servers on Linux x86_64 platforms when very large heap sizes are required. Contact Azul Systems to obtain and license Zing for LiveView. When Zing is installed in its default location, LiveView Server detects and launches with it by default. TIBCO LiveView servers support Azul Zing versions that correspond to Oracle JDK 8 versions.

In previous releases, StreamBase documentation described the opposite of hygienic modules as flexible modules. Starting with release 7.6.0, such modules are described as non-hygienic, which is a clearer term that needs no prior explanation.

The instruction page for integrating the m2e Eclipse Maven plug-in with StreamBase Studio was rewritten and corrected.

This release introduces a suite of JMS and TIBCO EMS adapters that replace the JMS and EMS adapters provided in previous releases. The new adapters are implemented as a suite of global Java operators, five for TIBCO EMS busses and five for JMS busses. The new operators store JMS configuration information in the StreamBase Studio project's primary .sbconf file, and no longer in independent .jmsconf files. The JMS Configuration Editor was updated to read from and write to the new .sbconf location, and can import settings from existing .jmsconf files. Features of the new adapters include:

The replaced JMS and EMS adapters remain part of StreamBase to support existing applications that use them, but are now deprecated. TIBCO strongly recommends migrating any StreamBase modules that connect to JMS or TIBCO EMS message servers to take advantage of the features of the new adapters.

The basic file reader and writer adapters included with StreamBase now each have a corresponding new adapter that can read from and write to files on HDFS-managed assets. The new adapters have the same feature sets as their corresponding regular file version, but with the addition of a User property setting. HDFS Adapters that have a control port can send a User setting through the control port. There is one exception about same features for corresponding adapters: the HDFS CSV File Reader does not have a tail mode, because that feature is not supported on HDFS. The eight new adapters are:

This release introduces a new adapter to interact with a TIBCO ActiveMatrix BPM server. The new adapter is implemented as a global Java operator, and wraps the ActiveMatrix BPM REST APIs to allow for any of the REST methods to be easily called from within a StreamBase application. The new adapter is described in Using the TIBCO ActiveMatrix BPM Operator and has a sample described in TIBCO ActiveMatrix BPM Operator Sample.

The Web Reader adapter has a new property, Maintain Line Separator. If enabled, this property maintains in the output result the newline and carriage return characters produced by the connected web server.

The IBM WebSphere MQ Input Adapter now supports the vendor's Read Ahead option. The Output adapter now supports asynchronous writes, when used with IBM MQ versions 7.0.1.9 and 7.1.0.2 or later.

This release includes updates to the API for creating custom file reading plug-ins to be used with the Data File option when defining Feed Simulations. The changes clarify the distinction between custom CSV file reading plug-ins and plug-ins that can read any file type, including binary files. The changes correct several inefficiencies in the previous API when using non-CSV reader plug-ins. The documentation page for this feature was updated, and the sample that illustrates custom plug-ins was reworked for clarity, with its documentation thoroughly rewritten.

The EventFlow Debugger gains the ability to evaluate simple expressions in the Eclipse Display view while debugging EventFlow modules. Such expressions can include numeric and string literals, operators, dynamic variables, constants, StreamBase expression language functions, and references to fields in the tuple flowing through the arc with the current breakpoint.

This release includes a new sample that illustrates the use of the TIBCO StreamBase® File Reader Adapter and File Writer Adapter working together to read a templated file into the system, then using the inbound tuple to replace values in the read file and then writing that information back out to another file. See File Template Sample.

As of TIBCO StreamBase release 7.5.4:

As of this release:

The TIBCO StreamBase® Adapter for Apache HBase is implemented as a suite of five global Java operators, including the HBase Admin, Delete, Get, Put, and Scan operators. See the HBase Operator Samples to learn how these operators can be used.

The OSI PI adapter was extended to support the ability to write to a connected PI database using the PI interface. Connections to both PI and AF servers are now disconnected and reconnected before each batch of requests is sent, to avoid potentially stale connections. And finally, reconnection attempts first try to use the last known good set of connection credentials, falling back to the adapter's Property view credentials as necessary.

The Metronome Operator was enhanced to accept a Maximum Tuples to Output integer. This allows a Metronome operator to emit a tuple periodically, then stop emitting altogether when the specified limit is reached.

In previous releases, you were required to use AS clauses to name fields when using a tuple() expression in both StreamSQL and LiveView's LiveQL. This restriction was lifted such that if you do not provide AS clauses and field names, generic field names are generated for you. For example, the following invocation of tuple(1, 2) generates the field names expr_0 and expr_1:

sbd --eval "tuple(1, 2)"
((expr_0 int, expr_1 int)) 1, 2

As of this release, the sbargen command has a new -w path option that sets the working directory of the sbargen command. When this parameter is specified, at runtime, sbargen sets its working directory before any other files or paths are read. When -w is used, all files and paths specified elsewhere on the sbargen command line MUST either be absolute paths OR be relative to the directory specified with -w. This includes all input and output files. At runtime, the specified path is checked for existence, and the command halts with an error if the path does not exist.

The Schema Compare dialog now defaults to ignoring field order.

The sample for the Web Reader adapter was updated and extended to clarify how to use the adapter in SOAP and RESTful web environments.

The TIBCO StreamBase® Output Adapter for TIBCO FTL® can now be configured to send tuples in batches instead of one at a time, and to specify a buffer flush interval time. This greatly improves the maximum throughput of the adapter.

The StreamBase Expression Language has the following new features as of this release:

The SchemaMismatchException class was moved into the Java Client API so that custom adapters have access to it without having to build against the Server runtime.

As of TIBCO StreamBase release 7.5.3:

The Copy Schema From dialog was extended to accept files in XML Schema format as well as CSV format. This allows you to create StreamBase named schemas by importing schemas saved from other TIBCO products such as TIBCO BusinessEvents, TIBCO BusinessWorks, and others. See Copying Schemas.

Release 7.3.2 quietly introduced the JSON input panel, an alternate way to send tuples to a stream in the Manual Input View. The current release extends the usability of the JSON input panel and of the Manual Input View in general with the following new features:

The JSON input panel and these new features are now documented on the Manual Input of Data page.

The content assistance provided by Studio when editing StreamBase expressions was improved. New features include:

When renaming a component on the EventFlow Editor canvas, you can now press Tab from the rename edit field of one component to jump to the next component in the downstream flow and open a rename edit window there.

The Schema Compare dialog now has a button that disables the automatic syncing of the scroll bars of the side-by-side windows.

Both StreamBase and Live Datamart applications can fail to compile, reporting an error similar to code attribute has grown beyond 64k limit. This primarily happens when those applications include very large Query Tables with more than one hundred columns. To work around such errors, you can specify the following system property for your application build processes. This property specifies the rewriting of generated code to eliminate the table size issue, then recompiling.

-Dstreambase.codegen.allow-refactoring=true

A new system property, streambase.querytable.load-initial-data, controls whether the initial data load feature of a Query Table's configuration is honored or not. The property's default state is true. Use this property when your application design needs to load initial table values during development, but needs to start with an empty Query Table for production deployment. In this case, set the property =false in the sbconf file for your production deployment configuration.

The release introduces two new global Java operators, the JSON to Tuple Converter and the Tuple to JSON Converter. These operators are invoked from the Adapters, Java Operators icon in the Palette view.

This release adds a pair of related adapters that provide a way for a StreamBase application to publish to and subscribe from an MQTT broker server. As described on this FAQ page, MQ Telemetry Transport is an international standard for machine-to-machine communication in the Internet of Things world of connected devices. The new adapters are described in MQTT Client Subscribe Adapter and MQTT Client Publish Adapter. The new adapters have a set of samples described in MQTT Client Publish And Subscribe Samples.

This release adds a pair of new adapters that connect to a PI database from vendor OSIsoft. The PI database ecosystem provides a way to manage real-time data and events in an enterprise's infrastructure, and is popular with oil and gas companies. Both new adapters require the OSI SDK, release 2012 or later, to be installed alongside StreamBase. Both adapters are input only and do not provide a way to write to an OSIsoft PI database. One new adapter called OSIPI uses OSIsoft's PiPoint mechanism to interact with a PI database, while the second adapter called OSIPIAF uses OSIsoft's Asset Framework mechanism. Both adapters are described on a combined page in the Adapters Guide, OSIsoft PI Adapters and have a simple sample.

The JMS Adapter was updated to support the Simple message type. The JMS Configuration File Editor was updated to support Simple messages, and the adapter's documentation was updated to match.

The IBM WebSphere MQ Input adapter was updated to support non-indexed queues. The adapter's Properties view has a new tab, Get Message Options, with two new properties, as described on IBM WebSphere MQ Input Adapter. The adapter also gained an option to allow you to specify the timeout value in milliseconds for an MQ get operation on the queue.

The StreamBase Exegy adapters now rely on version 3.5.0 or later of the JAR file that implements the Exegy Client API, XCAPI.jar. As before, this JAR file is provided as part of your Exegy installation and is not shipped with StreamBase.

The Shared Query Table sample was updated to provide a deployment file that starts the sample's three containers, and to also provide an sbbundle file as an illustration of using that feature.

As of TIBCO StreamBase release 7.5.2:

The StreamBase documentation for release 7.5.2 included the following changes:

This release adds a new Global Java Operator, the TIBCO Enterprise Runtime for R operator (the TERR operator), which allows StreamBase to use TIBCO's implementation of the R language to analyse and manipulate data. The operator requires an installation of TERR 2.7 or later on the same machine running StreamBase Server. The TERR operator is described on a new page of the Authoring Guide, and has a sample installed from File → Load StreamBase Sample as usual.

When specifying a declared schema for an Output Stream, the Properties View now has a check box labeled Do not fail typecheck for missing fields, which is cleared by default. Select this check box to allow the incoming stream's schema to be a subset of the declared schema. Incoming streams with a superset schema still fail typechecking.

This release adds two new functions and extends one function in the StreamBase Expression Language:

The Tuple to XML operator now has a new Pass Through Fields property that lets you specify an input field to convert to XML. The operator then copies the schema and values for all other fields to its output stream. When using this option, you must identify an XML Input Field Name for the tuple data to be converted to XML and can declare an XML Output Field Name to contain the XML document. The XML to Tuple operator also now these properties for use in the opposite conversion direction.

The TIBCO StreamBase® Web Server Request adapter now can parse parameters for the HTTP POST request "application/x-www-form-urlencoded" forms type and add the parsed data to the output. The adapter also now returns an HTTP ONLY field for cookies. Cookie fields were re-ordered to enable their direct streaming into the response side.

When setting a Roll Period of Daily for the TIBCO StreamBase® CSV File Writer adapter to close its output and open a new file, three new int properties (hour, minute, second) on the Rolling and Flushing tab of the Properties view now let you specify the exact time of day to perform the roll. Previously, all daily rolls occurred at midnight.

The TIBCO StreamBase® File Reader adapter can now replace tokens in non-binary input files with input tuple subfield values at run time. When the new Replacement Data Field Name property is selected, each token in an Input file is replaced by the contents of a subfield whose name matches the token. Specify tokens in the form of StreamBase parameters (${token-name}).

Tokens can include printf-style format strings enclosed in braces. For example, a double value might be specified in the input file by ${temp_celsius{%6.2f}}. There must be an input tuple subfield named temp_celsius to supply a value for the token, such as 26.4 as temp_celsius. Replacement is supported for all StreamBase data types, including tuples and lists. Tokens cannot be replaced by module parameters, only by properly-named fields defined for the adapter's input schema. For more information, see the Replacement Fields section of the File Reader Input Adapter page in the Adapters Guide.

The StreamBase documentation for release 7.5.1 included the following changes:

TIBCO Live Datamart 2.0.x is now automatically installed as part of TIBCO StreamBase® 7.5.x. As before, TIBCO Live Datamart is separately licensed. However, you no longer need to download and run a separate LiveView installer to access LiveView features.

The TIBCO StreamBase CEP Connectivity Package contains several premium adapters that have separate licensing concerns, and are not included in the base TIBCO StreamBase installation. If you licensed any of these adapters, install the StreamBase CEP Connectivity Package from the TIBCO download site.

The contents of this package are subject to change from release to release. Starting with release 7.5.0, the following adapter is no longer part of the Connectivity Package and now ships with both TIBCO StreamBase and TIBCO StreamBase Standard Edition:

If you have licensed this adapter as part of your StreamBase configuration, you no longer need to install the Connectivity Package to access it.