29West LBM Publishing Output Adapter

Contents

Introduction
Adapter Properties
Using the Adapter in a StreamBase Application
Typechecking and Error Handling
Suspend and Resume Behavior
Related Topics

Introduction

Note

This adapter is now sold as the TIBCO StreamBase® Adapter for Informatica Ultra Messaging.

The TIBCO StreamBase® Publishing Output Adapter allows a StreamBase application to publish LBM messages to one or more LBM topics. The adapter is embedded in the StreamBase application and has an input port that receives tuples to be published as LBM messages and an output port that emits status tuples. Each tuple enqueued to the adapter's input port causes a single LBM message to be published.

The adapter supports all StreamBase data types, including tuples and lists, though lists cannot contain elements of type list. Tuple fields of type tuple can be used to send nested LBM messages and LBM DECIMAL fields. The schema of the adapter's input port is determined by the output port of the upstream operator.

The adapter is configured through several properties set in the adapter's Properties view in StreamBase Studio. The properties include an optional configuration file to configure the LBM library, the input port field containing the topic to publish to, and the default topic value to use if the topic field is null.

Note

The 29West LBM Publishing Output adapter can consume significant system memory, depending on the data cache options you enable in the LBM configuration file. Memory allocated by the adapter is native memory that is allocated independent of (and competing for memory resources with) the JVM memory consumed by StreamBase Server. See Java VM Memory Settings for more on JVM versus native memory usage.

29West LBM Middleware Dependencies

This adapter requires access to the JAR file that implements the 29West LBM Java API on your system, and any files referenced by that JAR file. You will need to copy your site's LBM implementation version of that file, copy the file to $STREAMBASE_HOME/lib/ext.

Informatica purchased 29West and has since released an update of the LBM API, renamed Ultra Messaging Streaming Edition, or UMS. This adapter works with the UMS 6.x JAR files, but you cannot mix LBM and UMS installations. You must thoroughly uninstall any LBM 4.x kit before reinstalling a UMS 6.x kit. Since UMS is a workalike superset of LBM, this page uses "LBM" to refer to both.

The LBM JAR file, in turn, relies on various native library files, which are supplied as part of your LBM installation, and are not shipped with StreamBase. Make sure these files are locatable by the LBM JAR file on the PATH environment variable.

If you get an error message whose text refers to can't load lbmj: java.lang.UnsatisfiedLinkError: no lbmj in java.library.path, make sure the directory containing the LBM native libraries is on the path.

The LBM API implementation described in this section is a product of a third party, and its specifications and file names are subject to change by 29West. See your LBM documentation for the latest information.

Adapter Properties

Property Description
LBM Configuration File

The name of an optional LBM configuration file used to override the default settings used by the LBM library. Configuration files can be shared between one or more LBM subscribing and publishing adapter instances. To support such sharing, configuration files can have named sections, which are referenced by the LBM Configuration File Section property below. Named sections are identified by placing a section name inside square brackets within a comment:

#[MySection]

Lines that precede the first section name are considered part of the unnamed (default) section.

Lines that follow a section name are considered part of that section.

Section names must be unique within and across all referenced configuration files; a runtime error is flagged when a duplicate section name is encountered. Section names can include alphanumeric characters, digits, underscores, dots, and embedded spaces. They must start with an upper- or lower-case letter or an underscore.

LBM Configuration File Section The name of the LBM configuration file section from which to retrieve the parameters for configuring the LBMContext and LBMSource objects. If unspecified, parameters from the unnamed section are used. For enhanced performance, all LBM adapter instances that specify the same configuration file and section name share an LBMContext object.
Default Topic Contains the topic value under which to publish a LBM message if no topic field exists in the input schema, or its value is null in a specific input tuple.
Topic Field Name The name of the input port field containing the topic value under which to publish the LBM message. This field is optional if a Default Topic value is specified, which is used if no topic field exists, or its value is null in a specific input tuple.
Client Token Field Name The name of the input port field that can be used to send an arbitrary value that is returned in various publish status tuples.
Idle LBM Source Timeout The time, in seconds, to wait before an LBM source for an idle topic is discarded, or zero (the default) to disable the expiration of idle sources. An LBM source is created on demand when publishing to each new topic. When publishing to an unbounded number of topics, this property should be set to a non-zero value to limit the total number of LBM sources — each of which consumes finite resources — maintained by the adapter.
Capture Transform Strategy The strategy to use when transforming capture fields for this operator: FLATTEN or NEST.
Log Level Controls the level of verbosity the adapter uses to send notifications to the console. This setting can be higher than the containing application's log level. If set lower, the system log level is used. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE, and ALL.
Enable Pre-defined Messages

If disabled (the default), the LBM publish adapter sends self-describing messages, in which metadata describing the structure of the message is sent in each messsage. The LBM subscribe adapter uses this metadata to parse incoming messages and populate tuples.

If enabled, the LBM publish adapter sends pre-defined messages, in which the structure of the message is derived from the schema of the adapter's publish input port. The LBM subscribe adapter uses the schema of its message output port to parse incoming messages and populate tuples.

Schema Matching Policy

If Strict, the LBM publish and subscribe adapters must use identical schemas -- ones that contain the same fields in the same order -- to successfully exchange messages.

If BestVersion, the LBM publish and subscribe adapters must use compatible schemas to successfully exchange messages. Two schemas are compatible if their IDs match and the larger schema can be created simply by adding fields to the end of the smaller one.

Definition ID The message definition ID of the top-level schema used when exchanging pre-defined messages in BestVersion mode.
Major Version The major version of the message definition when using BestVersion schema matching.
Minor Version The minor version of the message definition when using BestVersion schema matching.
Nested Message Definition IDs The name and message definition ID of each tuple field in the output schema used when exchanging pre-defined messages in BestVersion mode.
Publish Type Overrides A table used to override the default LBM type used to publish specific fields when using self-describing messages. Each table row contains a StreamBase field name and the LBM type used to publish that field. StreamBase fields not present in the override table are published using the default LBM type specified in the table below.

See the Publish Type Overrides Table below for the correspondence between StreamBase types, default LBM types, and valid overrides.

Use Fully-Qualified StreamBase Field Names When enabled, nested StreamBase fields are specified in the Publish Type Overrides table using fully-qualified names. Otherwise, simple field names relative to the enclosing tuple are used. Use fully-qualified names when a top-level and nested field with the same relative name need to be published using different LBM types.
Enable Persistence If enabled (not the default), the adapter registers with a UME store and recovers from its previous position in the message stream after a failure.
UME Session ID Session ID used by the Ultra Messaging library to manage recovery.
UME Store Name Name of UME store as defined in the context-name option of the umestored's XML configuration file.
Sequence Number Retrieval Ports If enabled (not the default), the adapter creates an additional input and output port to retrieve the retrieve the current sequence number for a topic.
UME Store Notifications If enabled (not the default), the adapter emits status tuples in response to UME store registation events.
Reciver Connection Notifications If enabled (not the default), the adapter emits status tuples when receives connect and disconnect to topics.
Message Sequence Number Information If enabled (not the default), the adapter emits status tuples to convey the sequence numbers with which messages are sent.
Message Delivery Confirmations If enabled (not the default), the adapter emits status tuples when messages are delivered to receivers.
Message Stability Notifications If enabled (not the default), the adapter emits status tuples when messages become durable in the UME sture.
Message Reclaimed Notifications If enabled (not the default), the adapter emits status tuples when messages are reclaimed.
Flight Size Notifications If enabled (not the default), the adapter emits status tuples when the number of in-flight messages exeeds or falls below the configured threshold.

The following table shows, for each StreamBase type, the default LBM type, and the valid override values that can be used to publish the StreamBase field using a non-default LBM type.

Publish Type Overrides Table

StreamBase Type Default LBM Type Valid Overrides
blob BLOB —
bool BOOLEAN —
double DOUBLE FLOAT
int INT32 INT8, INT16, UINT8, UINT16
long INT64 UINT32, UINT64
string STRING UNICODE
timestamp TIMESTAMP —
tuple MESSAGE DECIMAL
list<blob> BLOB ARRAY —
list<bool> BOOLEAN ARRAY —
list<double> DOUBLE ARRAY FLOAT ARRAY
list<int> INT32 ARRAY INT8 ARRAY, INT16 ARRAY, UINT8 ARRAY, UINT16 ARRAY
list<long> INT64 ARRAY UINT32 ARRAY, UINT64 ARRAY
list<string> STRING ARRAY UNICODE ARRAY
list<tuple> MESSAGE ARRAY DECIMAL ARRAY

Using the Adapter in a StreamBase Application

This section discusses how to use the 29West LBM Publishing Output Adapter in a StreamBase application. As shown in the diagram below depicting one of the adapter's sample applications, the publishing adapter has one primary input and one primary output port to communicate with the surrounding application. When configured for persistence, the publishing adapter provides an additional input port and output port to retrieve a topic's current message sequence number.

The 29West LBM Publishing Output Adapter's ports are used as follows:

  • Publish: Tuples enqueued on this input port cause the adapter to publish LBM messages, one per tuple. The port's schema is derived from the output port of the upstream operator and can contain all StreamBase data types, including lists and nested tuples to provide hierarchy in LBM messages. Tuple fields are mapped to LBM message fields by name. List fields can contain all StreamBase element types, except lists. Fields containing null in the tuple are discarded; no corresponding field is placed in the LBM message.

    DECIMAL fields are published using a tuple field of type tuple containing two sub-fields: a long for the mantissa and an int for the exponent. The two sub-fields may appear in either order in the nested tuple field. If a field to be published as a DECIMAL contains null values in a sub-field, a warning is emitted, the tuple field is discarded, no DECIMAL field is placed in the LBM message. If both sub-fields are null, the tuple field is discarded silently.

  • ReadSequenceNumber: This optional port is used to retrieve a topic's current message sequence number and is enabled by setting the Sequence Number Retrieval Ports property on the Persistence tab. Tuples enqueued on this input port cause the adapter to publish LBM messages, one per tuple. Each tuple enqueued to this port results in a tuple emitted on the SequenceNumberOut port. The ReadSequenceNumber port has the following schema:

    • Topic, string: Contains the name of the topic for which to retrieve the current sequence number.

    • ClientToken, string: An arbitrary value that is echoed in the response, allowing the StreamBase application to associate requests with responses.

  • PublishStatus: This output port emits status, information, and error tuples. The PublishStatus port has the following schema:

    • type, string: Contains one of the following values describing the type of event that occurred:

      • Transport

      • Process Tuple

      • Process LBM Message

      • Subscription

      • Source Event

      • Suspend/Resume

    • object, string: the name of the object associated with the event, such as the name of the tuple field that couldn't be converted to an LBM field or the name of the adapter that was suspended or resumed.

    • action, string: the action associated with the event, such as a conversion error or adapter suspended or resumed.

    • message, string: A human-readable message string.

  • SequenceNumberOut: This output port emits tuples containing the current sequence number for a topic in response to request tuples enqueued to the ReadSequenceNumber port. The SequenceNumberOut port has the following schema:

    • Topic, string: Contains the name of the topic for which the current sequence number has been retrieved.

    • ClientToken, string: The value placed in the corresponding field of the ReadSequenceNumber request tuple.

    • SequenceNumber, long: The current sequence number for the topic, or null if no sequence number is available.

Add an instance of the adapter to a new StreamBase application as follows:

  1. In StreamBase Studio, create a project, and create an EventFlow application file to host the adapter.

  2. Drag the Output Adapter icon from the Operators & Adapters drawer in the Palette view to the canvas. Select the 29West LBM Publishing Output Adapter from the list and click OK.

  3. Connect an input and output stream to the adapter's input and output ports, respectively. Configure the schema of input stream with the fields required in the LBM messages to be published. Include a Topic field of type string if the default value for the Topic Field Name property is to be used.

  4. Select or double-click the adapter icon, and in the Properties view, click Adapter Settings and fill in any desired properties. Either a Topic Field Name or Default Topic must be specified.

Typechecking and Error Handling

The 29West LBM Publishing Output adapter uses typecheck messages to help you configure the adapter in your StreamBase application. In particular, the adapter generates typecheck messages when:

  • The schema of the input port contains one or more list fields with elements of type list.

  • An invalid LBM configuration file is specified.

  • An invalid topic field name within the input port schema is specified.

  • The topic name within the input port schema is not of type string.

  • Neither a topic field not a default topic is specified.

  • An invalid client token field name within the input port schema is specified.

  • The client token name within the input port schema is not of type string.

  • Persistence is enabled and no UME sesion ID is specified.

  • Persistence is enabled and an invalid UME sesion ID is specified.

  • Persistence is enabled and no UME store name is specified.

  • The datatypes of one or more fields in the input schema are incompatible with the specified LBM override types.

The adapter generates messages on the status port during runtime under various conditions, including:

  • The topic field of a publish tuple is null and no default topic has been configured.

  • An error occurs populating an LBM field from a tuple field.

  • An error is returned by the LBM library when the adapter attempts to publish an LBM message.

  • The adapter is suspended or resumed.

Suspend and Resume Behavior

When suspended, the 29West LBM Publishing Output Adapter stops processing input tuples and emitting output tuples.

When resumed, the adapter begins processing input tuples and emitting output tuples once again.

Related Topics