Intermediate Stream Dequeuing

For debugging purposes only, you can dequeue data from intermediate streams in your application, not only from output streams. This debugging feature allows you to examine the value of data at an intermediate point in the application before it has been fully processed and before it has arrived on an output stream. This feature is not to be used for inserting new data into the middle of an application.

Intermediate stream dequeuing (ISD) is disabled by default. You can enable it for a particular run or debug session as described below in Enabling ISD.

Do not confuse intermediate stream dequeuing with the always expose feature of input and output streams.

For another debugging approach, consider using runtime tracing in place of, or in addition to, ISD. Runtime tracing is described in Runtime Tracing and Creating Trace Files, and includes a comparison table in Tracing Versus Dequeuing.

The following table clarifies what streams are exposed for dequeuing for various settings of intermediate stream dequeuing.

You can enable intermediate stream dequeuing in several ways:

With all methods, you can enable either full or partial ISD:

Enable ISD in StreamBase Studio

Follow these steps to expose intermediate streams when running or debugging an application in Studio:

1. Configure and save a run or debug configuration for the application whose intermediate streams you want to expose. Run configurations are described in Editing Launch Configurations.

2. On the Advanced tab of the run configuration dialog, check the Enable Intermediate Stream Dequeue check box. When used without a filter pattern, this specifies full ISD.

3. Optional step, to specify partial ISD. To restrict the number of intermediate streams exposed, enter a regular expression pattern to match against stream names. Intermediate streams whose name matches the provided expression as a substring are exposed for dequeuing; all other intermediate streams are not.

4. Run the application by running the launch configuration. Intermediate streams now appear as selectable output streams in the Application Output view's list of streams.

The following illustration shows the Output Stream Selector dialog from the Application Output view for two cases of running the Bollinger Band sample installed with StreamBase.

• You see the dialog on the left when running with the default run configuration. It shows only the three output streams defined in the application.

• You see the dialog on the right when running with full intermediate stream dequeuing enabled. It shows the same three output streams plus the output port of all intermediate components as selectable streams.

sbd --intermediate-stream-dequeue BestBidsAsks.sbapp

<sysproperty 
    name="streambase.codegen.intermediate-stream-dequeue" 
    value="true" />

<sysproperty 
    name="streambase.codegen.intermediate-stream-dequeue-regex" 
    value="pattern" />

<sysproperty 
    name="streambase.codegen.intermediate-stream-dequeue-regex" 
    value="Map\\d" />

With intermediate stream dequeuing enabled, you can dequeue data from any output port of any operator using a stream name of the form:

out:operatorName_N

where operatorName is the name of the operator and N is the number of the output port for that operator.

Consider the Split.sbapp application, which is one of the operator samples installed with the StreamBase kit:

To verify the intermediate stream names, run sbc list when the application is running with ISD enabled. For example:

container   default
container   system
stream      INTRUSION_TooManyIPsForUser
stream      INTRUSION_TooManyUsersForIP
stream      IPandUserLogin
stream      out:CheckIPsForAUser_1
stream      out:CheckUsersInAnIP_1
stream      out:ProcessIPFirst_1
stream      out:ProcessIPFirst_2
schema      schema:IPandUserLogin
operator    CheckIPsForAUser
operator    CheckUsersInAnIP
operator    IPCountExceeded
operator    ProcessIPFirst
operator    UserCountExceeded

Compare the EventFlow diagram above with the sbc list output. Notice the following:

You can also determine intermediate stream names by viewing the XML source for your EventFlow application. Look for the stream values associated with output ports. For example:

<box name="ProcessIPFirst" type="split">
  <input port="1" stream="IPandUserLogin"/>
  <output port="1" stream="out:ProcessIPFirst_1"/>
  <output port="2" stream="out:ProcessIPFirst_2"/>
  <param name="output-count" value="2"/>
...

Intermediate stream names can be found that do not appear to follow the naming convention described in the previous section. When determining the name of intermediate streams from which to dequeue, you must confirm the names as actually used in your application. Confirm with an sbc list command or by viewing the XML source of your EventFlow application.

In some cases, the name of an intermediate stream can vary from the default convention, depending on the history of edits to the application. For example, consider following installed sample, AggregateByDim.sbapp:

When you run this sample in debug mode (or with JVM argument set as above), an sbc list command returns:

container  default
container  system
stream     AvgPricePSOut
stream     OutputStream1
stream     TradesIn
schema     schema:TradesIn
operator   Aggregate2Dimensions
operator   ConvertTimeToSeconds

By default, the intermediate stream between the Aggregate2Dimensions and ConvertTimeToSeconds operators would have been named out:Aggregate2Dimensions_1. But at some point in this application's history, the Aggregate2Dimensions operator was connected to an output stream named OutputStream1, and was subsequently disconnected from that output stream (which was either removed or renamed).

Despite the edits, the original output stream name (OutputStream1) defined for Aggregate2Dimensions is still in use. In other words, do not assume that each operator's output stream name always follows the default convention.

For related information, see:

Back to top ^