StreamBase to StreamBase Output Adapter

Introduction

The TIBCO StreamBase® SBD-to-SBD Output (Upstream) Adapter runs within an upstream application, sending tuples to a downstream application. It is suitable for situations in which multiple data producers — upstream applications — wish to enqueue data to a single consumer. The producers can be stopped and started independently without affecting the consumer.

The adapter is configured with the host name and port of the downstream application as well as the name of the stream, or streams to send data to. The adapter is configured by default to reconnect after a restart of the downstream application.

Adapter Properties

The following table explains the adapter properties that you can set in the Adapter Settings tab of the Properties view for an SBD2SBDOutput icon in the EventFlow canvas.

Property	Data Type	Description
Downstream host	string	The host name or IP address of the downstream application.
Downstream port	string	The TCP port number of the downstream application.
Downstream stream	string	The name of the first stream to write to in the downstream application. If no stream with this name is present in the downstream application, or the stream's schema is incompatible with the schema configured in the adapter, the upstream application behaves as though the downstream application is inaccessible. The Reconnect Interval property determines whether the upstream application attempts to reconnect periodically until a compatible stream is present. To specify the names of the second and subsequent streams to write to, enter the stream names in separate rows in the Additional Streams tab of the Properties view.
Downstream container	string	The name of the container in the downstream application that holds the Downstream stream designated in the previous field. The default entry is `default`.
Reconnect Interval (secs)	int	The period, in seconds, the upstream application waits between reconnection attempts after the downstream application fails or no compatible stream is present. Set this property to zero to disable reconnection; the default value is 10 seconds. When reconnection is disabled, the downstream application must be started first and must have a compatible output stream from which the upstream application can read. Note When specifying a reconnection interval as part of a container connection to a remote host, specify the interval value in milliseconds, not seconds. See Remote Container Connection Parameters.
Enqueue buffer size (tuples)	int	The numbers of tuples to enqueue before tuples are sent to the downstream application. For a low-latency application, set the buffer size to zero.
Max enqueue buffer size (tuples)	int	The maximum size of the enqueue buffer before tuples are dropped. Enter `-1` to disable this setting.
Enqueue flush interval (msecs)	integer	The flush interval of the enqueue buffer in milliseconds. For a low-latency application, set the flush interval to a small non-zero number. Do not set it to zero, which causes the adapter to perform a busy loop and consume excessive CPU cycles.
Use SSL	check box	Select this check box if the connection to the downstream host uses secure SSL-based communication to the downstream server.
Authentication type	drop-down list	Specifies the type of authentication to use when connecting to the downstream server. Options in the drop-down list are: `No authentication`, `Basic authentication`, and `Client certificate authentication`.
User name	string	For secure connections, the login name on the downstream host to authenticate as. Only one of User name or Keystore pathname can be specified.
Keystore file pathname	string	The path on the local machine to the file containing keystore information used for Client certificate authentication.
Password	string	The password for the User name or the keystore passphrase.
Schema matching	two-option radio buttons	Identical structure required means that, in the schemas of the outgoing and incoming streams, each field must match in data type and sequence, but the field names do not have to match. For fields of type tuple, their fields must also match in type and sequence. Match field names, the default setting, means that fields in the outgoing stream are matched by name against fields in the incoming stream. Fields whose names match must have the same data type in both streams. Any fields in the outgoing schema whose names don't match the incoming schema are not streamed. Any fields in the incoming schema whose names don't match anything in the incoming schema are set to null. For example, consider an outgoing stream with schema `(a int, b (x int, y int, z int))` that is connected to an incoming stream with schema `(b (y int, z int), c int)`. The incoming stream does not see fields `a` or `b.x`. Fields `b.y` and `b.z` are passed from incoming to outgoing, and field `c` in the incoming stream is set to null.
Capture Transform Strategy	radio button	The strategy to use when transforming capture fields for this operator: FLATTEN or NEST.
Explicit URI (Advanced)	string	This option is for advanced users who have a standard StreamBase URI they use in other parts of the StreamBase application. Use this option to paste in a full StreamBase URI to the downstream application, including security connection information. If this field is used, the following fields must be blank: Downstream host, Downstream port, Downstream stream, Downstream container, User name, Keystore file pathname, and password.
Enable event port	check box	Select this check box to enable an output port that receives connection-up and connection-down events. The schema of the event output port is as follows: EventType — `Connection` or `Status` Object — For EventType `Connection`, this field contains the URL of the upstream server instance. For EventType `Status`, this field contains the adapter canvas name. Action — contains UP or DOWN to indicate the state of the connection with the upstream server. Info — contains a human-readable description of the event, such as `Connection to sb://localhost:10001 is UP`
Connect on adapter init	check box	Selected by default. When selected, or no control port is enabled, the adapter attempts to connect to the other server during adapter start-up. When cleared, the adapter connects to the other server when a connect tuple is received on its control input port.
Enable control port	check box	Cleared by default. When selected, an additional input port is created, used to control and request status of the adapter. See below for more on this input port
Low Latency	check box	Cleared by default. When selected, the adapter configures itself to minimize data latency by disabling buffering, Nagle algorithms, and so on.
Log Level	Drop-down list	Controls the level of verbosity the adapter uses to issue informational traces to the console. This setting is independent of the containing application's overall log level. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE, and ALL.

Configuration Settings for Highest Throughput and Lowest Latency

Low Latency, Enqueue buffer size and Enqueue flush interval are three properties that affect the latency and throughput of the StreamBase to StreamBase output adapter.

Enabling Low Latency configures the adapter in its lowest latency mode. Enqueue buffer size and Enqueue flush interval are ignored in this mode.

For maximum throughput, set the Enqueue buffer size property large enough to reduce the per-packet overhead to a minimum. Values of 10 to 1000 are common; the default is 100. Set the Enqueue flush interval property high enough so that, in general, the enqueue buffer size is reached, and the data is sent, before the flush interval expires. For high throughput applications, the flush interval is rarely hit.

For minimum latency, set the Enqueue buffer size property to 0. This forces each tuple to be sent immediately, in line. With a buffer size of zero, the Max enqueue buffer size and Enqueue flush interval properties are not used. This results in the lowest latency configuration, but comes at the cost of higher system overhead. Tuples are sent one per network packet, which increases network overhead and increases CPU usage on both the client and server. For low latency applications, see Java System Properties for more information on configuring clients and servers.

Setting the Enqueue buffer size property to 1 or higher causes tuples to be placed on a queue, and a network writer thread performs the actual network writes. The latency difference between an Enqueue buffer size of 0 and 1 is only the cost of adding the tuple to a queue and notifying the write thread. The disadvantage of an Enqueue buffer size of 0 is that if the downstream dequeuer is slow, the application must pause when network backup occurs, as the network write occurs in line.

Adapter Port Schemas

Output ports

The StreamBase to StreamBase output adapter has an optional event output port defined above.

Downsteam Input Ports

For the default stream, and each additional configured stream, an input port is added to the adapter.

Control Input Port

If you select the Enable control port option, an input port is added to the adapter for use as a control port. The first field of the control port schema must be a string field named command. This port is used to send the following commands to the adapter:

connect — The adapter tries to connect to the configured downstream host. If configured to reconnect, it continues to try to connect at the configured reconnect interval until it succeeds.
disconnect — The adapter disconnects from the downstream host. It does not try to reconnect, even if a reconnect interval is configured.
status — The adapter emits a status tuple on the event port, if configured.

Typechecking and Error Handling

Typechecking fails if the Downstream Host or Downstream Stream properties are empty or if the Downstream Port property contains an illegal TCP port number (unless the Explicit URI field is used instead).

The behavior in response to errors depends upon the setting of the Reconnect Interval property. When configured to reconnect, the upstream application polls periodically until the downstream application is accessible and contains an output stream with an identical name and schema. Otherwise, the upstream application emits an error message and makes no further attempts to connect to the downstream application.

Dropped Tuple Warning Behavior

The StreamBase to StreamBase output adapter throttles dropped-tuple warnings as follows:

The first dropped tuple generates an immediate warning.
Subsequent dropped tuples are counted, but not immediately reported.
When at least one tuple has been dropped, and at least 10 seconds have elapsed since the previous dropped-tuple warning, a new warning is generated the next time a tuple is processed, whether that tuple is dropped or not.
Dropped-tuple warnings indicate the number of tuples dropped and the number of seconds elapsed (normally 10) since the last warning, or in the case of the first warning, since the adapter was started.
No warnings are generated if no tuples are dropped since the last warning.
A warning can cover more than 10 seconds if the adapter drops a tuple and then becomes idle.

Suspend and Resume Behavior

When suspended, the adapter stops sending tuples to the downstream application. Note that doing so may cause the application to drop the connection with the downstream application, resulting in lost tuples. Upon resuming, the adapter reconnects to the downstream application, if necessary and configured to do so, and begins sending tuples. If the adapter is configured not to reconnect with the downstream application and the connection is dropped during the suspension, the application will not enqueue tuples to the downstream application upon resume.

However, when the container enqueue (dequeue) control is set explicitly to ENABLED from any other state (DISABLED, DROP_TUPLES, or even ENABLED), the StreamBase to StreamBase adapter then starts (that is, enters the STARTED state), regardless of the adapter's Start with application setting.