Contents
The applications in this sample group demonstrate the use of the TIBCO StreamBase® Adapter for Thomson Reuters Velocity Analytics Broadcast Server, the TIBCO StreamBase® Adapter for Thomson Reuters Velocity Analytics UTSS, and query operations against the Velocity Analytics time series data store.
The Broadcast Server input adapter receives real-time market data from the Velocity Analytics Broadcast Server via the real-time multicast interface, converts each tick into a StreamBase Tuple, and sends it to the engine for processing. The UTSS adapter inserts data into a user time series store (UTSS) via the Analytics Engine Firebird SQL interface.
Familiarity with the Thomson Reuters Enterprise Platform for Velocity Analytics is required. In addition, a Velocity Analytics development server environment is required with the following installed:
-
Analytics Engine
-
Broadcast Server
The following Velocity Analytics documentation, available on the Thomson Reuters Customer Zone, provides additional information about Velocity Analytics, its components, data formats, and the SQL interface to use to query the time series database and User Time Series Store (UTSS).
-
Installation Guide — provides instructions on installing Velocity Analytics components.
-
System Admin and Config Guide — provides additional information about data record formats.
-
Hardware and Software Guide — provides information about current hardware and operating system requirements.
-
Using SQL with Velocity Analytics — provides information about querying the Velocity Analytics time series bases and UTSS.
The set of Velocity Analytics samples includes the following EventFlow applications. These applications can be executed within StreamBase Studio or using the StreamBase Server command line utility from a terminal window on UNIX or StreamBase Command Prompts on Windows.
-
BSInputAdapter.sbapp
— demonstrates ingesting ticks from the real-time multicast interface and dequeuing them to simple output streams based on record type. -
UTSSOutputAdapter.sbapp
— demonstrates accepting filtered Trade and Quote records from the Broadcast Server adapter and subsequent writing them to a UTSS. -
UTSSSelect.sbapp
— demonstrates submitting a query to the target UTSS. This application uses UTSSSelectModule.sbapp, itself an EventFlow application, as a module to handle administrative interface details to the UTSS. -
VWAP.sbapp
— demonstrates calculating the VWAP analytic on the event stream. It is an end-to-end application from multicast ingestion to UTSS output. -
TimeConversionFunctions.sbapp
— demonstrates the use of custom time conversion functions. -
QueryAnalyticsEngine.sbapp
— demonstrates the use of the StreamBase Query operator to retrieve data from the Analytics Engine time series data store using standard JDBC access.
In StreamBase Studio, import this sample with the following steps:
-
From the top menu, select
→ . -
Select
Thomson Reuters Velocity Analytics
from the StreamBase Market Data Adapters category. -
Click OK.
StreamBase Studio creates a project for this sample.
When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.
Important
Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.
Using the workspace copy of the sample avoids the permission problems that can occur when trying to work with the initially installed location of the sample. The default workspace location for this sample is:
studio-workspace
/sample_adapter_embedded_thomsonreuters-velocityanalytics
See Default Installation Directories for the location of studio-workspace
on your system.
In the default TIBCO StreamBase installation, this sample's files are initially installed in:
streambase-install-dir
/sample/adapter/embedded/thomsonreuters-velocityanalytics
See Default Installation Directories for the default location of studio-workspace
on your system.
Some of the applications in this sample require multicast Trade and Quote data from the Broadcast Server. This data can be supplied from a market data feed or by configuring the Broadcast Server to replay a sample feed log included in the distribution (recommended). The sections below, which document how to use each sample, assume you are using replay data from the sample feed log.
This procedure configures the Broadcast Server to replay market data from the sample feed log provided in the Velocity Analytics distribution at the following location:
install_dir
\Samples\Data\Feed_20070301.log
This file contains trade and quote records for March 1, 2007 within the time range 10:08:47 AM and 10:11:30 AM for the following symbols:
AAPL.O, AMGN.O, DELL.O, EBAY.O, GOOG.O, INTC.O, MSFT.O, ORCL.O, PFE.N, T.N
-
Install the Velocity Analytics Broadcast Server, following the instructions in the Velocity Analytics Installation Guide.
-
Configure playback mode. On the system hosting the Broadcast Server, open the system configuration file,
, in a text editor.install_dir
\Config\TBAdmin.ini-
Locate the
playbackmode
configuration parameter and set it to 1, as follows:playbackmode=1
-
Locate the following parameters in the
broadcastserver
process section and note their settings. They are used when configuring the EventFlow applications.listenip groupip portnoofstock
-
Confirm the following setting in the FeedReplay Configuration section of
TBAdmin.ini
:FeedName=
install_dir
\Samples\Data\Feed_20070301.log
Save and close
TBAdmin.ini
. -
-
Start the Broadcast Server. From a Windows command prompt, type:
net start broadcastserver
This initiates feed file playback.
-
Confirm feed replay has successfully started. Look at the Broadcast Server message log,
install_dir
\Log\BSMessages.log, for a message similar to the following:INFO, 635711,SendP: 1st packet: 200......@@a,FR,INTC.O, Trade,....
-
From a Windows command prompt, type:
net start searchengine
-
Confirm the Analytics Engine successfully started. Look at the Analytics Engine message log,
install_dir
\Log\SEMessages.log, for a message similar to the following:INFO,6000018,VA Init: Initialization successfully completed: INFO,190142,SETclThreads waiting for accepts with 13 threads
-
Confirm the Analytics Engine has successfully synchronized with the Broadcast Server. Look at the Analytics Engine message log,
install_dir
\Log\SEMessages.log, for a message similar to the following:INFO,190931, ****** Synch record found in file - switching to MC stream INFO,190932, ****** Synch record: 201...,@@a,FR,EBAY.O,Trade,,...,1947080,...
This sample demonstrates ingesting ticks from the Velocity Analytics real-time multicast interface provided by the Broadcast Server and dequeuing them to simple output streams based on record type. It requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.
-
In the Package Explorer, double-click to open the
BSInputAdapter.sbapp
application. -
Specify the Broadcast Server:
-
Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.
-
Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.
Adapter Property Broadcast Server Configuration Parameter Local IP listenip
Multicast IP groupip
Multicast Port portnoofstock
-
-
Click the Run button. This opens the SB Test/Debug perspective and starts the application.
-
In the Test/Debug Perspective, open the Application Output view. If connectivity to your Broadcast Server is configured correctly, look for tuples emitted in the Application Output view indicating the adapter has connected successfully.
-
When done, press F9 or click the Stop Running Application button.
This application receives real-time trade and quote records from the Velocity Analytics Broadcast Server adapter. The components
TradeFieldSelection
and QuoteFieldSelection
filter out the fields that are not acceptable for insertion into a UTSS data store. Trade and Quote records for MSFT.O
with BidPrice
> 0.0 are filtered and written into an UTSS archive using the adapter. Rejected trades and quotes are sent to RejectedMSFTOTrades
and RejectedMSFTOQuotes
output streams respectively.
This sample requires the Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.
-
In the Package Explorer, double-click to open the
UTSSOutputAdapter.sbapp
application. -
Specify the Broadcast Server:
-
Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.
-
Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.
Adapter Property Broadcast Server Configuration Parameter Local IP listenip
Multicast IP groupip
Multicast Port portnoofstock
-
-
Specify the UTSS database:
-
Select the Velocity Analytics UTSS adapter icon to open the Properties view for the adapter.
-
Select the Adapter Properties tab and enter valid values for UTSSPath and UTSSName.
-
-
Start the Analytics Engine. Refer to Starting the Analytics Engine for details.
-
Click the Run button. This opens the SB Test/Debug perspective and starts the application.
-
Select the Manual Input tab and enter the command
connect
to the ControlStream input stream. Click Send Data to send the command. -
In the Test/Debug Perspective, open the Application Output view. If connectivity to your Broadcast Server and UTSS database is configured correctly, look for tuples emitted on the Statistics stream in the Application Output view, which indicates that records have been written.
-
When finished, issue a
disconnect
command to the Control port of the Broadcast Server adapter. When all buffered data has been processed, press F9 or click the Stop Running Application button.
This application demonstrates the use of the StreamBase Query operator to retrieve data from a User Time Series Store (UTSS).This
sample requires an existing UTSS database, such as created using the
application.
UTSSOutputAdapter.sbapp
-
Configure the JDBC data source. In the Package Explorer, double-click
sbd.sbconf
. Change the IP address in the Analytics Engine data source definition to that of the Analytics Engine hosting the UTSS. This is the same address as defined with thelistenip
configuration parameter in thesearchengine
process section of the Velocity Analytics system configuration file,TBAdmin.ini
.Save your changes.
-
Specify the UTSS data base. In the Package Explorer view, double-click UTSSSelect.sbapp. Select the Parameters tab and:
-
Enter values for UTSSPath and UTSSName. These values are those used when the UTSS was created via
UTSSOutputAdapter.sbapp
. -
Enter
Quote
for UTSSTable.
-
-
Review the query. In the Package Explorer, double-click to open the
UTSSSelectModule.sbapp
application.-
Select the SelectFromUTSSTable Query operator icon to open its Properties view.
-
Select the Query Settings tab. From this tab you can review the SQL statement to be submitted to the UTSS.
-
Select the Result Settings tab. Make sure
Quote_AE
is selected as the private schema name.
-
-
Run the application. In the Package Explorer view, double-click to open the
UTSSSelect.sbapp
application. Click the Run button. This opens the SB Test/Debug perspective and starts the application. -
In the Test/Debug Perspective, click the Manual Input tab. Enter the desired symbol, plus startTime and endTime values. Times are entered with the format
yyyymmddhhmmss.ttt
symbol MSFT.O
startTime A start time well before the time you started the UTSSOutputAdapter.sbapp
application.endTIme An end time after you stopped the UTSSOutputAdapter.sbapp
application. -
Click the Send Data button.
-
In the Test/Debug Perspective, open the Application Output view. If connectivity to your UTSS is configured correctly, look for tuples emitted in the Application Output view indicating the adapter has connected successfully.
-
When finished, press F9 or click the Stop Running Application button.
This EventFlow application demonstrates the use of the Velocity Analytics UTSS Output Adapter to store a VWAP analytics output in a User Time Series Store (UTSS). It receives real-time records from a Velocity Analytics Broadcast Server Input adapter. In this application, the Broadcast Server Input Adapter is configured to emit only Trade records. The Field Selection component filters out fields that are not needed for VWAP calculations. VWAP is then calculated on the resultant output with a window of 10 seconds. The result is grouped by symbol.
The VWAP output from the aggregate operator contains openTime
and closeTime
fields in StreamBase timestamp format. Since a UTSS database understands only VHTime, the StreamBase timestamps are converted
to VHTime using the time conversion function sbTimestampToVH()
. This function is implemented in BSInputAdapter.jar
(and also in UTSSOutputAdapter.jar
) and can be invoked using StreamBase's calljava()
function. After the time is converted, the output is sent to the Velocity Analytics UTSS Output Adapter for subsequent write
to the configured UTSS archive.
This sample requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.
-
In the Package Explorer , double-click
VhayuRecordDefinitions.xml
. If not already present, add the following flex record definition near the beginning of the file, within the<FRDict>
tag.<FRDef name="VWAP" defid="44001"> <field name="NumTrades" type="t_u32" /> <field name="OpenTime" type="t_VHTime" /> <field name="CloseTime" type="t_VHTime" /> <field name="VWAP" type="t_double" /> </FRDef>
For your convenience, this definition is also included in the file,
SampleFRDefinitions.xml
, available via the Package Explorer. -
In the Package Explorer, double-click to open the
VWAP.sbapp
application. -
Specify the Broadcast Server:
-
Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.
-
Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.
Adapter Property Broadcast Server Configuration Parameter Local IP listenip
Multicast IP groupip
Multicast Port portnoofstock
-
-
Specify the UTSS database:
-
Select the Velocity Analytics UTSS adapter icon to open the Properties view for the adapter.
-
Select the Adapter Properties tab and enter valid values for UTSSPath and UTSSName.
-
-
If not already running, start the Analytics Engine. Refer to Starting the Analytics Engine for details.
-
Click the Run button. This opens the SB Test/Debug perspective and starts the application.
-
Select the Manual Input tab and enter the command
connect
to the ControlStream input stream. Click Send Data to send the command. -
In the Test/Debug Perspective, open the Application Output view. If connectivity to your Broadcast Server and UTSS data base is configured correctly, look for tuples emitted on the VWAP Output stream in the Application Output view.
-
When finished, issue a
disconnect
command to the Control port of the Broadcast Server adapter. When all buffered data has been processed, press F9 or click the Stop Running Application button.
This sample demonstrates the use of various time conversion functions to reformat timestamps. It requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.
-
In the Package Explorer, double-click to open the
TimeConversionFunctions.sbapp
application. -
Specify the Broadcast Server:
-
Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.
-
Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.
Adapter Property Broadcast Server Configuration Parameter Local IP listenip
Multicast IP groupip
Multicast Port portnoofstock
-
-
Click the Run button. This opens the SB Test/Debug perspective and starts the application.
-
In the Test/Debug Perspective, open the Application Output view. If connectivity to your Broadcast Server is configured correctly, look for tuples emitted on the Application Output stream for the various types of time conversion formats.
-
When finished, issue a
disconnect
command to the Control port of the Broadcast Server Input adapter. When all buffered data has been processed, press F9 or click the Stop Running Application button.
This application demonstrates the use of StreamBase JDBC Query operator to retrieve data from the Analytics Engine time series data store using SQL. This sample requires the Analytics Engine to be installed, started, and loaded with data.
If not already running, start the Analytics Engine. Refer to Starting the Analytics Engine for details.
-
Configure the JDBC data source. In the Package Explorer, double-click
sbd.sbconf
. Change the IP address in the Analytics Engine data source definition to that of the Analytics Engine. This is the same address as defined with thelistenip
configuration parameter in thesearchengine
process section of the Velocity Analytics system configuration file,TBAdmin.ini
.Save your changes.
-
In the Package Explorer view, double-click
QueryAnalyticsEngine.sbapp
. -
Select the Query Settings tab. From this tab you can review the SQL statement to be submitted to the Analytics Engine.
-
Click the Run button. This opens the SB Test/Debug perspective and starts the application.
-
In the Test/Debug Perspective, click the Manual Input tab. Enter the desired symbol, plus startTime and endTime values. Times are entered with the format
.yyyymmddhhmmss.ttt
-
Click the Send Data button.
-
In the Test/Debug Perspective, open the Application Output view. If connectivity to your Analytics Engine is configured correctly, look for tuples emitted on the output stream indicating the adapter has connected successfully.
-
When finished, press F9 or click the Stop Running Application button.
This section describes how to run the samples in UNIX terminal windows or Windows command prompt windows. On Windows, be sure to use the StreamBase Command Prompt from the Start menu as described in the Test/Debug Guide, rather than the Windows default command prompt.
This sample demonstrates ingesting ticks from the Velocity Analytics real-time multicast interface provided by the Broadcast Server and dequeuing them to simple output streams based on record type. It requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.
-
Open six terminal windows on UNIX, or six StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.
-
In window 1, type:
sbd -f sbd.sbconf BSInputAdapter.sbapp
-
In window 2, type:
sbc dequeue -v Trade
-
In window 3, type:
sbc dequeue -v Quote
-
In window 4, type:
sbc dequeue -v Statistics
-
In window 5, type:
sbc dequeue -v Status
This window displays tuples dequeued from adapter's output port for Status information.
-
Observe that tuples representing
Trade
data,Quote
data, andStatistics
information are being emitted from window 2, window 3, and window 4 respectively. -
In window 6, type:
echo status | sbc enqueue ControlStream
-
In window 6, type the following command to terminate the server and the dequeuers:
sbadmin shutdown
This application receives real-time trade and quote records from Velocity Analytics Broadcast Server adapter. Trade and Quote
records for MSFT.O
with BidPrice
> 0.0 are filtered and written into an UTSS archive using the adapter.
This sample requires the Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.
-
Open five terminal windows on UNIX, or five StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.
-
In window 1, type:
sbd -f sbd.sbconf UTSSOutputAdapter.sbapp
-
In window 2, type:
echo connect | sbc enqueue ControlStream
This establishes a connection between the UTSS Output Adapter and the Analytics Engine.
-
In window 3, type:
sbc dequeue -v Status
-
In window 4, type:
echo status | sbc enqueue ControlStream
-
In window 5, type:
sbc dequeue -v Statistics
Observe the emitted tuples in the same window. The tuples carry information about SQL inserts being applied to the specified UTSS archive. Use any Firebird SQL client tool to confirm that the data is continuously written to the UTSS archive while the application is running. See the Velocity Analytics guide, Using SQL with Velocity Analytics, available in the Velocity Analytics documentation set for more information about UTSS databases.
-
In window 4, type the following command to terminate the server and the dequeuers:
sbadmin shutdown
This application demonstrates the use of a StreamBase Query operator to retrieve data from a User Time Series Store (UTSS).
This sample requires an existing UTSS data base, such as created using the
application. If not already running, start the Analytics Engine. Refer to Starting the Analytics Engine for details.
UTSSOutputAdapter.sbapp
-
Open three terminal windows on UNIX, or three StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.
-
In window 1, type:
sbd -f sbd.sbconf UTSSSelect.sbapp
-
In window 2, type:
sbc dequeue -v SelectOutput
This window displays the query result set in form of StreamBase tuples.
-
In window 3, type:
echo MSFT.O, 20010101, 20020101 | sbc enqueue QueryParameters
Observe the query result set in window 2.
-
In window 3, type the following command to terminate the server and the dequeuers:
sbadmin shutdown
This EventFlow application demonstrates the usage of the Velocity Analytics UTSS Output Adapter to store an analytics output (VWAP) in a User Time Series Store (UTSS). It receives real-time records from Velocity Analytics Broadcast Server Input Adapter. In this application, the Broadcast Server Input Adapter is configured to emit only Trade records. The VWAP is then calculated on the resultant output with a window of 10 seconds. The result is grouped by symbol.
The VWAP output from the aggregate operator contains openTime
and closeTime
fields in StreamBase timestamp format. Since a UTSS database understands only VHTime, the StreamBase timestamps are converted
to VHTime using the time conversion function sbTimestampToVH()
. This function is implemented in BSInputAdapter.jar
(and also in UTSSOutputAdapter.jar
) and can be invoked using StreamBase's calljava()
function. After the time is converted, the output is sent to the Velocity Analytics UTSS Output Adapter for subsequent write
to the configured UTSS archive.
This sample requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.
-
Open four terminal windows on UNIX, or four StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.
-
In window 1, type:
sbd -f sbd.sbconf VWAP.sbapp
-
In window 2, type:
echo connect | sbc enqueue ControlStream
This establishes a connection between the UTSS Output Adapter and the Analytics Engine.
-
In window 3, type:
sbc dequeue -v VWAPOutput
This window displays the VWAP calculation output.
-
Wait until the time window for the VWAP calculation (default 10 seconds) expires and observe the VWAP output in window 3. The same data is also written to the specified UTSS archive in UTSS output adapter properties. Use any Firebird SQL client tool to confirm that the data is continuously written to the UTSS archive while the application is running. See Using SQL with Velocity Analytics, available in the Velocity Analytics documentation set, for more information about UTSS databases.
-
In window 4, type the following command to terminate the server and the dequeuers:
sbadmin shutdown
This sample demonstrates the use of various time conversion functions to reformat timestamps. It requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.
-
Open seven terminal windows on UNIX, or seven StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.
-
In window 1, type:
sbd -f sbd.sbconf TimeConversionFunctions.sbapp
-
In window 2, type:
sbc dequeue -v GMT_ASCII
This window displays the time field in ASCII format adjusted for GMT time zone.
-
In window 3, type:
sbc dequeue -v LOCAL_ASCII
This window displays the time field in ASCII format adjusted for local time zone.
-
In window 4, type:
sbc dequeue -v SERIALTIME
This window displays the time field in the Serial time format.
-
In window 5, type:
sbc dequeue -v SBTIMESTAMP
This window displays the time field in the Serial time format.
-
In window 6, type:
sbc dequeue -v VHTIME
This window displays the time field in VHTime format.
-
Observe the time values in GMT_ASCII, LOCAL_ASCII, SERIAL, SBTIMESTAMP, VHTIME format in windows 2,3,4,5, and 6 respectively.
-
In window 7, type the following command to terminate the server and the dequeuers:
sbadmin shutdown
This application demonstrates the use of a StreamBase JDBC Query operator to retrieve data from the Analytics Engine time series data store using SQL. This sample requires the Analytics Engine to be installed, started, and loaded with data.
If not already running, start the Analytics Engine. Refer to Starting the Analytics Engine for details.
-
Open three terminal windows on UNIX, or three StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.
-
In window 1, type:
sbd -f sbd.sbconf QueryAnalyticsEngine.sbapp
-
In window 2, type:
sbc dequeue -v OutputStream
This window displays the query result set in the form of StreamBase Tuples.
-
In window 3, type:
echo MSFT.O, 20070101, 20080101 | sbc enqueue QueryParameters
-
Observe the query result set in window 2.
-
In window 3, type the following command to terminate the server and the dequeuers:
sbadmin shutdown