Contents
These samples illustrate how to use the Python operator, and how to configure a local Python execution environment with the Python Instance operator.
The Python operator allows you to execute any valid Python code within an EventFlow module. The Python operator and its companion Python Instance operator allow Python-centric teams to reuse existing Python code in an event processing context without requiring major rewrites to that code. The Python operators allow the execution of Python-based statistical modeling, data science processing, and machine learning produced with Python packages such as SciPy and TensorFlow.
In StreamBase Studio, import this sample with the following steps:
-
From the top-level menu, select
→ . -
Enter
python
to narrow the list of options. -
Select Using the Python operator from the Large Data Storage and Analysis category.
-
Click
.
StreamBase Studio creates a single project containing the sample files.
Before running the samples, you must edit two configuration files to specify the full path to the Python executable of your installed and preferred version of Python:
-
Edit
src/main/resrouces/adapter-configurations.xml
to specify the Python version for the global instance of Python to use in EventFlow modules as a whole. The file affects the P01-Version sample. -
Edit
src/main/configurations/sbengine.xml
to specify the Python version for local instances defined by Python Instance operators. This file affects the P02-Local-Instance and P03-TensorFlow samples.
Example Python executable paths for supported Python versions are shown in Python Versions Supported in the StreamBase Authoring Guide.
This sample demonstrates performing a very basic Python call and returning a result.
-
Make sure you have edited this sample's
adapter-configurations.xml
configuration file to specify the path to the Python version you want to use for the global instance. -
In the Project Explorer view, open the sample you just loaded.
If you see red marks on a project folder, wait a moment for the project to load its features.
If the red marks do not resolve themselves after a minute, select the project, right-click, and select
→ from the context menu. -
Open the
src/main/eventflow/
folder.packageName
-
Open the
P01-version.sbapp
file and click the Run button. This opens the SB Test/Debug perspective and starts the module. -
The sample runs itself one time on startup by means of a Once adapter. Look in the Output Streams view for a report of the configured Python source and version number.
-
To run the sample again, click Manual Input view. This sends an empty tuple that populates another row in the Output Streams view.
in the -
When done, press F9 or click the Terminate EventFlow Fragment button.
This sample demonstrates two things:
-
Asking Python to perform a simple calculation of the Fibonacci numbers that are less than a provided integer.
-
Making that Python calculation in a local instance defined by a Python Instance operator.
Note
On macOS, this sample runs with pypy or pypy3 only if you set LANG=en_US.UTF-8
in the Environment tab of the Run Configuration for this module. (Or set LANG to the appropriate value for your locale.)
Follow these steps:
-
Make sure you have edited this sample's
sbengine.xml
configuration file to specify the path to the Python version you want to use for this sample's local instance. -
In the Project Explorer view, open the sample you just loaded.
If you see red marks on a project folder, wait a moment for the project to load its features.
If the red marks do not resolve themselves after a minute, select the project, right-click, and select
→ from the context menu. -
Open the
src/main/eventflow/
folderpackageName
-
Open the
P02-local-instance.sbapp
file and click the Run button. This opens the SB Test/Debug perspective and starts the module. -
In the Manual Input view, the
InputStream
stream is preselected.Enter any integer into the
fib
field, such as100
,3000
, or11000
.Click
. -
In the Output Streams view, look for an OutputStream tuple that show a list of the members of the Fibonacci series that are less than the integer you sent.
-
Enter another integer in the
fib
input field and press . -
Now, in the Manual Input view, select the Input Stream named
zControl
. -
Enter
stop
in the command field and press . -
Select the
InputStream
stream again, and send another integer to Python. This time, the result is an Adapter Exception error message.This is because the
stop
command disabled the local Python execution environment defined by the Python Instance operator. Notice that the name of this operator isThisInstance
, and notice that in the Properties view for the Python operator that its Instance Type is Local and its Local Instance Id isThisInstance
.This means the Python operator is configured to operate in the local execution environment named
ThisInstance
, but in steps 7 and 8, you disabledThisInstance
. -
Re-enable the local execution environment by selecting the
zControl
input stream and sending the commandstart
. -
Now return to the
InputStream
stream and send in more integers, which succeed this time. -
When done, press F9 or click the Terminate EventFlow Fragment button.
This sample demonstrates using Google's open source TensorFlow machine learning framework to run a model in multiple thread instances, to evaluate model input, and to receive model output. This sample requires that the Python installation you wish to use is configured with the TensorFlow and OpenCV-Python packages. In addition, there are limitations on the Python vendors and versions you can use with TensorFlow and with this sample.
The S03-TensorFlow sample was tested with TensorFlow 1.6.0 and 1.7.0 on Windows, macOS, and Linux. On all platforms, the 1.7.0 release shows Console view warnings about deprecated modules, but otherwise, the two versions run the same.
The TensorFlow package, and also therefore this sample, has the following limitations:
-
On Windows, TensorFlow requires Python 3.5 or 3.6. It is not supported with Python 2.7 on Windows.
-
On Windows, ActiveState provides both Python 3.5 and 3.6 installers, but recommends using their Python 3.5 installer.
-
The P03-TensorFlow sample does not run with ActiveState Python 3.6.
-
ActiveState Python 3.5 installs a large set of data science Python packages, including TensorFlow and OpenCV-Python. However, these bundled packages are old versions and must be upgraded with commands like the following:
pip3 install --update tensorflow pip3 install --update opencv-python
-
-
On macOS and Linux, the TensorFlow web site claims support for TensorFlow as a Python 2.7 package. However, the pip commands associated with Python 2.7 on these platforms do not locate or download the TensorFlow packages. You may be able to manually install TensorFlow for use with Python 2.7.
-
This sample does not run with Pypy because no Pypy-compatible TensorFlow package is available.
The usual way to install Python packages is with the pip or pip3 command. Make sure you have installed a compatible pip command to match your Python version and vendor, as described in Python Versions Supported in the StreamBase Authoring Guide.
With standard installations of Python and Pip as described in that section, Windows and macOS users should need only the following commands:
pip3 install tensorflow pip3 install opencv-python
On Linux, preface the same commands with sudo -H
:
sudo -H pip install tensorflow sudo -H pip install opencv-python
Use the pip or pip3 command associated with your Python version to confirm that these packages were installed for that Python version:
pip3 list
or
pip list
The P03-TensorFlow sample uses a worker module to allow eight operations to run simultaneously. The worker module is run with
a StreamBase concurrency setting of multiplicity=8
. Each simultaneous module has its own instance of Python and initializes the model within each instance. This occurs in worker-only.sbapp
, which does not run on its own, but only when called by P03-tensorflow.sbapp
.
-
In the Project Explorer view, open the sample you just loaded.
If you see red marks on a project folder, wait a moment for the project to load its features.
If the red marks do not resolve themselves after a minute, select the project, right-click, and select
→ from the context menu. -
Open the
src/main/eventflow/
folder.packageName
-
Open the
P03-tensorflow.sbapp
file and click the Run button. This opens the SB Test/Debug perspective and starts the module, which is self-running by means of a Once adapter. -
In the Output Streams view, select the stream named
OutputStream
in the Stream control. After a moment of operation, this stream shows a list of image files and their result strings. -
If you wish to rerun the model processing, send any tuple (including a null tuple) on the
Restart
input stream, and click . -
When done, press F9 or click the Terminate EventFlow Fragment button.
When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.
Important
Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.
Using the workspace copy of the sample avoids permission problems. The default workspace location for this sample is:
studio-workspace
/sample_python
See Default Installation Directories for the default location of studio-workspace
on your system.