Contents
This sample describes how to use the dataAreaPath
configuration property in a HOCON configuration file, and how to make use of the Operator.getDataDirectory()
method call in the StreamBase Java Client API.
This sample's src/main/configurations
folder contains the sbengine.conf
file, whose purpose is to define a custom location for the data directory used by all StreamBase fragments that run from
the same StreamBase engine. See the StreamBase Configuration Guide for more information regarding how to use configuration files.
The default value for the dataAreaPath
property in the sbengine.conf
file is a folder named engine-data-area
. This folder is placed in the node directory for the node that runs this EventFlow fragment. When nodes are installed and
run in StreamBase Studio, node directories are placed in the .nodes
folder of the current Studio workspace. The default node directory name is nodename.clustername
, where the node name is constructed from a pattern defined in > or in a Run Configuration.
The default cluster name is your system login name, which is also changeable in Studio's Preferences. The default data directory
is further placed in a folder named fragments
, and then in a folder named for the fully qualified name of the current EventFlow module, with periods replaced by underlines.
Without specifying a dataAreaPath
name-value pair in a configuration file, this sample would write its data files into the .nodes/sample_javaoperator_datadir.sbuser/fragments/com_tibco_sb_sample_javaoperator_datadir_datadir0/engine-data-area/
folder of the current Studio workspace.
The datadir.sbapp
EventFlow module uses two custom Java operators:
-
DataDirectoryFileWriter writes files as specified on the module's FileIn input stream into the data directory specified in the
sbengine.conf
file. -
DataDirectoryObserver reads and lists the files that exist in the specified data directory.
You can specify the dataAreaPath
property with either a relative or absolute path. Relative paths are relative to the default location of engine-data-area
as described above. If your configuration file specifies:
dataAreaPath = "myDataDir"
then your files are stored in the same long path shown above, with myDataDir
replacing the name engine-data-area
.
If you specify: dataAreaPath = "../myDataDir"
then the myDataDir
folder is placed as a subfolder under fragments
.
If you specify: dataAreaPath = "../../myDataDir"
then myDataDir
is placed at the root of your node directory, and so on.
This sample's configuration file shows:
dataAreaPath = "../../../../sample_javaoperator-datadir/myDataDir"
which places myDataDir
at the root of this Studio project if it was loaded with the Import Samples and Community Content dialog. You may need to use > on this sample's project folder to see the newly created folder.
Use the following variation if you copied this sample's folder to another location or if you are running outside of Studio:
dataAreaPath = "../../../../javaoperator-datadir/myDataDir"
If you run this sample as part of a StreamBase application with the epadmin command on the command line, you specify a node directory name and location with the nodedirectory=
option. If unspecified, the current directory of the epadmin command is used. If you specify a nodedirectory
option of ~/tmp/nodedirs
for a node, you start with nodename A.sbuser, and the default location for the data directory becomes:
~/tmp/nodedirs/A.sbuser/application/engines/default-engine-for-com.tibco.sb.sample.javaoperator-datadir/engine-data-area/
Finally, you can specify an absolute path such as dataAreaPath = "/tmp/myDataDir"
or dataAreaPath = "C:/tmp/myDataDir"
.
However, any absolute path is likely to be machine-specific or operating system-specific, which limits the ability to run your EventFlow fragment in multiple nodes for High Availability purposes. If your need to use an absolute path, consider using a substitution variable with a default value, so that the data directory path can be specified at deployment time. Substitution variables are described in the StreamBase Configuration Guide.
For example:
dataAreaPath = "${DATADIR:-/tmp/myDataDir}"
In StreamBase Studio, import this sample with the following steps:
-
From the top-level menu, select
> . -
Enter
data dir
to narrow the list of options. -
Select Specifying a custom data directory from the Extending StreamBase category.
-
Click
.
StreamBase Studio creates a project for the sample.
To run this sample:
-
In the Project Explorer view, open the sample you just loaded.
If you see red marks on a project folder, wait a moment for the project to load its features.
If the red marks do not resolve themselves after a minute, select the project, right-click, and select
> from the context menu. -
Open the
src/main/eventflow/
folder.packageName
-
Open the
datadir.sbapp
file and click theRun button. This opens the SB Test/Debug perspective and starts the module.
-
Select the Manual Input view. Select
from the Input Stream list. Use FileIn to inject tuples that have two string fields, fileName and fileContent.The DataDirectoryFileWriter operator writes to the specified data directory a file with the specified name and contents, first prepending
DDFWOperator_
to the file name you specify. The operator outputs a tuple containing the number of files in the data directory and the most recent write in the data directory. -
Input a few file names, clicking
after each one. -
Select the
input stream to inject a no-fields tuple to force the DataDirectoryObserver operator to read the data directory.This operator then emits the number of files in the data directory as well as a list of tuples. Each tuple contains a filename and either the entire first line or the first 50 characters of the corresponding file, whichever is shorter.
-
When done, press F9 or click the
Terminate EventFlow Fragment button.
When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.
Important
Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.
Using the workspace copy of the sample avoids permission problems. The default workspace location for this sample is:
studio-workspace
/sample_javaoperator-datadir
See Default Installation Directories for the default location of studio-workspace
on your system.