Using the Feed Simulation Editor

Use the Copy from Stream/Named Schema button to quickly copy the schema from an existing stream or named schema in your workspace.

Clicking this button opens a dialog that shows you a tree list of all modules and interfaces in your current Studio workspace. Click the arrows next to the project folder that you know contains the module or interface of interest. Select the module or interface that contains the stream or named schema of interest. Select that stream or schema and click OK to copy that schema to the Simulation Streams section.

Let's say a StreamBase application has two input streams: TradesIn and FuturesIn. In the Simulation Streams section, click New Stream to invoke the Add Simulation Stream dialog. In the following example, we have identified the name of the additional input stream in the application:

In this dialog, add a schema to the feed simulation using either of the following methods:

After clicking OK twice, the updated Simulation Streams section looks like this:

For Boolean fields, a number with absolute value 0.5 or greater is considered true. For Boolean fields, you can select:

For fields of type list, the Generation Method column offers the ListSize option. You can specify the minimum and maximum number of elements to be generated, which defaults to 4 elements for both fields.

For fields of type tuple, each field of the tuple is shown separately. Each field's row offers the appropriate choice for its data type, as described in previous sections.

For all data types, in the Probability of generating null option, you can specify a range or 0.0 to 1.0 to indicate how often to randomly set the field's value to null. For example, entering 0.12 causes the field's value to be null in about 12% of the generated tuples. By default, this field contains the value 0, meaning the feed simulation will not set any field values to null.

This topic shows you how to specify conditional field generation in the Customize Fields dialog.

The following example is based on the feed simulation for the StreamBase Loop sample (loop-feedsim.sbfs) that you can open and modify. The sample's schema is {symbol: string, price: double, qty: int}. Let's assume that you want to make the price field conditional on the value of symbol. When you first declare a field to be conditional, a dialog displays for you to select the field(s) it will depend on, as the following steps describe.

Notice that the Generation Method for symbol is set to RandomString. This is not helpful for any fields that depend on symbol, because at runtime they would rarely if ever key to a symbol value and thus their values would default.

It is much better to enumerate values for symbol, making them predictable and specific to the purpose of the simulation. For this simulation example, a limited set of actual stock symbols could could be itemized, such as AAPL, MSFT, IBM, and ORCL. To do that:

The grid for symbol now looks like this:

The price field is already conditional on the symbol field, so select price again to begin specifying how it should depend on each particular symbol.

After you have specified the generation for price based on the AAPL symbol, you can save time by clicking the Duplicate button to clone that entry on the next row down. Use the drop-down symbol menu to select a different symbol, and then fill in the random value parameters appropriately for that symbol. The Customize Fields dialog for price now looks like this:

You can complete the simulation setup by selecting field qty and making it also depend on the symbol field's values. If you want to use random values, the procedure is the same as for specifying conditional prices, except that a random walk of ints will be generated for the qty field.

Click OK to dismiss the Customize Fields dialog. Save your modified feed simulation as loop-feedsim-conditional.sbfs before running it.

In the Select Fields dialog you can choose more than one field to control how values are generated for a conditional field. For instance, suppose you create a feed simulation for an application that computes weekly average and extreme temperatures from daily weather data for a set of cities at different times during the year. By making the ranges of random temperatures depend on both city name and month, you can generate data appropriate to each location and time period by using actual weather records to set the range of temperatures that the simulation will generate.

When you make field values depend on more than one field, you need to specify combinations of outcomes. That is, you describe a generation rule for each possible combination of specified values of several other fields. In the temperature data simulation just described, to specify temperature ranges for every month of the year, you would need to enumerate 12 sets of them for each city for which you want to provide data, in addition to providing a default value for each.

Here is an example of a Customize Fields dialog in which temperature is based on five cities and two months of the year, which produces ten cases. If all 12 months were included, there would be 60 cases to specify.

A random value is generated for the AvgTemp field (daytime average temperature for a given day) for each of these ten cases, based on parameters specified for each one. For example, the Boston-January combination has parameters: Minimum: -5° F, Maximum: 40° F, Start: 30° F, Increment 1° F, as shown below:

These parameters are independent of those for any other city-month combination, although you are free to duplicate cases without making changes.

A conditional field can depend on values generated for another conditional field. The only restriction is that you must not introduce circular dependencies. That is, if field B is conditional on field A and field C is conditional on field B, field A cannot depend directly or indirectly on either B or C.

As long as the values you select to identify cases are well-specified, you can use a sequence of conditional fields to create something like a decision tree. For instance, in the previous example that generates temperature data, the City field was enumerated and not conditional. If the schema also had a state identifier string (e.g., AK ... WY), you could make the City field conditional on it, and for each state enumerate a set of cities that it contains. Temperature and windspeed data would still be generated for specific combinations of city and month, and cities would be properly assigned to states. Were cities not conditional, they would be randomly associated with states.

In theory, you could continue to chain location identifier fields like this all the way down to individual weather stations. However, the number of cases would grow geometrically if you choose to specify all their hierarchical combinations. To avoid having to create long lists of such conditional values, keep the number of enumerated values small or allow unspecified combinations to default.

To learn more, see the Feed Simulation Conditional Field Sample documentation and run the sample application and feed simulation in StreamBase Studio.

The Timestamp from Column feature is discussed in Using the Timestamp from Column Feature.

The Include in Synchronized Timestamp Group feature is discussed in Using Synchronized Timestamp Groups.

With the Prefill and loop on Tuple buffer feature enabled, the feed simulation reads from the specified data file or JDBC data source. The tuple buffer grows to fit the data source, until either all data source rows have been read, or the buffer reaches the limit specified in the Tuple buffer field, or the process runs out of memory (which results in an error). The feed simulation then sends tuples from the tuple buffer from beginning to end. When the last tuple is reached, the feed simulation immediately resends the first tuple in the buffer, then resends subsequent tuples according to the Data rate and other settings for this simulation.

If you also specify the Timestamp from column option, the buffer is sent according to the timing information in the timestamp column. When replaying the tuple buffer, the second and subsequent times through the buffer are also replayed with the same timing.

As tuples are resent from the tuple buffer, no fields are modified in the buffer, including any timestamp or sequence number fields. Let's say the first tuple read into the buffer has sequence number 1, and the last tuple has sequence number 20,000. On starting over and rereading the buffer, sequence number 1 is resent to the running StreamBase application. If your application requires monotonically increasing sequence numbers or timestamp fields, then you must adjust the application logic to detect the restart of a looped-buffer feed simulation, to compensate for the restart of sequenced fields. For example, your application's input stream logic might add 20,000 to each sequence number after a loop restart is detected.

You can use this technique to generate a larger feed simulation data set from a smaller one. For example, let's say you captured a day's worth of trading data in real time in a CSV data file or JDBC database. You want to use this data set with its real-world timestamps to run a month's worth of trading data for testing or benchmarking purposes. To do so, use the Prefill and loop on Tuple buffer option, and have your application's input logic detect when the last tuple in the buffer is received. Then add one day to the received timestamp of the next set of tuples received.

StreamBase handles empty strings encountered in incoming data files, database queries, or feed simulations with custom generated data as follows:

You can also set a specific string to be treated as an incoming null field value, as described in the Null string field in the preceding section.