Python Operator Samples

Contents

About the Samples
Importing This Sample into StreamBase Studio
Samples Setup
Running the P01-Version Sample
Running the P02-Local-Instance Sample
Running the P03-TensorFlow Sample
Running the P04-Script-Resource Sample
Sample Location

About the Samples

These samples illustrate how to use the Python operator, and how to configure a local Python execution environment with the Python Instance operator.

The Python operator allows you to run any valid Python code within an EventFlow module. The Python operator and its companion Python Instance operator allow Python-centric teams to reuse existing Python code in an event processing context without requiring major rewrites to that code. The Python operators allow the execution of Python-based statistical modeling, data science processing, and machine learning produced with Python packages such as SciPy and TensorFlow.

Importing This Sample into StreamBase Studio

In StreamBase Studio, import this sample with the following steps:

  • From the top-level menu, select File>Import Samples and Community Content.

  • Enter python to narrow the list of options.

  • Select Using the Python operators from the Large Data Storage and Analysis category.

  • Click Import Now.

StreamBase Studio creates a single project containing the sample files.

Samples Setup

Before running the samples, you must edit two configuration files to specify the full path to the Python executable of your installed and preferred version of Python:

  • Edit src/main/configurations/Python.conf to specify the Python version for the global instance of Python to use in EventFlow modules as a whole. The file affects the P01-Version sample.

  • Edit src/main/configurations/sbengine.conf to specify the Python version for local instances defined by Python Instance operators. This file affects the P02-Local-Instance, P03-TensorFlow, and P04-Script-Resource samples.

Example Python executable paths for supported Python versions are shown in Python Versions Supported in the StreamBase Authoring Guide.

Running the P01-Version Sample

This sample demonstrates performing a very basic Python call and returning a result.

  1. Make sure you have edited this sample's Python.conf configuration file to specify the path to the Python version you want to use for the global instance.

  2. In the Project Explorer view, open this sample's folder.

    Keep an eye on the bottom right status bar of the Studio window. Make sure any Updating, Downloading, Building, or Rebuild project messages finish before you proceed.

  3. Open the src/main/eventflow/packageName folder.

  4. Double-click to open the P01-version.sbapp module. Make sure the module is the currently active tab in the EventFlow Editor.

  5. Click the Run button. This opens the SB Test/Debug perspective and starts the module.

  6. As the server starts, StreamBase Studio switches to the SB Test/Debug perspective.

  7. The sample runs itself one time on startup by means of a Once adapter. Look in the Output Streams view for a report of the configured Python source and version number.

  8. To run the sample again, click Send Data in the Manual Input view. This sends an empty tuple that populates another row in the Output Streams view.

  9. When done, press F9 or click the Terminate EventFlow Fragment button.

Running the P02-Local-Instance Sample

This sample demonstrates two things:

  • Asking Python to perform a simple calculation of the Fibonacci numbers that are less than a provided integer.

  • Making that Python calculation in a local instance defined by a Python Instance operator.

Note

On macOS, this sample runs with pypy or pypy3 only if you set LANG=en_US.UTF-8 in the Environment tab of the Run Configuration for this module. (Or set LANG to the appropriate value for your locale.)

Follow these steps:

  1. Make sure you have edited this sample's sbengine.conf configuration file to specify the path to the Python version you want to use for this sample's local instance.

  2. In the Project Explorer view, open this sample's folder.

    Keep an eye on the bottom right status bar of the Studio window. Make sure any Updating, Downloading, Building, or Rebuild project messages finish before you proceed.

  3. Open the src/main/eventflow/packageName folder.

  4. Double-click to open the P02-local-instance.sbapp module. Make sure the module is the currently active tab in the EventFlow Editor.

  5. Click the Run button. This opens the SB Test/Debug perspective and starts the module.

  6. As the server starts, StreamBase Studio switches to the SB Test/Debug perspective.

  7. In the Manual Input view, the InputStream stream is preselected.

    Enter any integer into the fib field, such as 90, 500, or 1100.

    Click Send Data.

  8. In the Output Streams view, look for an OutputStream tuple that show a list of the members of the Fibonacci series that are less than the integer you sent.

  9. Enter another integer in the fib input field and press Send Data.

  10. Now, in the Manual Input view, select the Input Stream named zControl.

  11. Enter stop in the command field and press Send Data.

  12. Select the InputStream stream again, and send another integer to Python. This time, the result is an Adapter Exception error message.

    This is because the stop command disabled the local Python execution environment defined by the Python Instance operator. Notice that the name of this operator is ThisInstance, and notice that in the Properties view for the Python operator that its Instance Type is Local and its Local Instance Id is ThisInstance.

    This means the Python operator is configured to operate in the local execution environment named ThisInstance, but in steps 7 and 8, you disabled ThisInstance.

  13. Re-enable the local execution environment by selecting the zControl input stream and sending the command start.

  14. Now return to the InputStream stream and send in more integers, which succeed this time.

  15. When done, press F9 or click the Terminate EventFlow Fragment button.

Running the P03-TensorFlow Sample

This sample demonstrates using Google's open source TensorFlow machine learning framework to run a model in multiple thread instances, to evaluate model input, and to receive model output. This sample requires that the Python installation you wish to use is configured with the TensorFlow and OpenCV-Python packages. In addition, there are limitations on the Python vendors and versions you can use with TensorFlow and with this sample.

The S03-TensorFlow sample was tested with TensorFlow 1.13.1 on Windows, macOS, and Linux.

TensorFlow Limitations

The TensorFlow package, and also therefore this sample, has the following limitations:

  • On Windows, TensorFlow requires Python 3.5 or 3.6. It is not supported with Python 2.7 on Windows.

  • On Windows, ActiveState provides both Python 3.5 and 3.6 installers, but recommends using their Python 3.5 installer.

    • The P03-TensorFlow sample does not run with ActiveState Python 3.6.

    • ActiveState Python 3.5 installs a large set of data science Python packages, including TensorFlow and OpenCV-Python. However, these bundled packages are old versions and must be upgraded with commands like the following: 

      python3 -m pip install --update tensorflow
python3 -m pip install --update opencv-python

  • On Linux and macOS, TensorFlow supports Python 2.7, 3.4, 3.5, or 3.6.

  • This sample does not run with Pypy because no Pypy-compatible TensorFlow package is available at the time of this writing in April, 2019.

  • This sample is tested on python version 3.6

Installing TensorFlow and OpenCV-Python Packages

The usual way to install Python packages is with the pip or pip3 command. Make sure you have installed a compatible pip command to match your Python version and vendor, as described in Python Versions Supported in the StreamBase Authoring Guide.

With standard installations of Python and Pip as described in that section, Windows and macOS users should need only the following commands: 

pip3 install tensorflow
pip3 install opencv-python

On Linux, preface the same commands with sudo -H: 

sudo -H pip install tensorflow
sudo -H pip install opencv-python

If TensorFlow is installed, upgrade to the latest version: 

sudo -H pip install --upgrade tensorflow
sudo -H pip install --upgrade opencv-python

Use the pip or pip3 command associated with your Python version to confirm that these packages were installed for that Python version: 

pip3 list

or 

pip list

Steps to Run

The P03-TensorFlow sample uses a worker module to allow eight operations to run simultaneously. The worker module is run with a StreamBase concurrency setting of multiplicity=8. Each simultaneous module has its own instance of Python and initializes the model within each instance. This occurs in worker-only.sbapp, which does not run on its own, but only when called by P03-tensorflow.sbapp.

  1. Make sure you have edited this sample's sbengine.conf configuration file to specify the path to the Python version you want to use.

  2. In the Project Explorer view, open this sample's folder.

    Keep an eye on the bottom right status bar of the Studio window. Make sure any Updating, Downloading, Building, or Rebuild project messages finish before you proceed.

  3. Open the src/main/eventflow/packageName folder.

  4. Double-click to open the P03-tensorflow.sbapp module. Make sure the module is the currently active tab in the EventFlow Editor.

  5. Click the Run button. This opens the SB Test/Debug perspective and starts the module, which is self-running by means of a Once adapter.

  6. As the server starts, StreamBase Studio switches to the SB Test/Debug perspective.

  7. In the Output Streams view, select the stream named OutputStream in the Stream control. After a moment of operation, this stream shows a list of image files and their result strings.

  8. If you wish to rerun the model processing, send any tuple (including a null tuple) on the Restart input stream, and click Send Data.

  9. When done, press F9 or click the Terminate EventFlow Fragment button.

Running the P04-Script-Resource Sample

This sample demonstrates three ways to load a Python script:

  • From a Python file.

  • From the script text box.

  • Enable the control port and load a script while running.

Follow these steps:

  1. Edit this sample's sbengine.conf file to specify the path to the Python version you want to use.

  2. In the Project Explorer view, double-click to open P04-script-resource.sbapp. Make sure the application is the currently active tab in the EventFlow Editor.

  3. Double-click the Python adapter icon to see its Properties view. Open the Script tab. By default, the Script source control specifies Script text.

  4. Save the module and click the Run button. This opens the SB Test/Debug perspective and starts the application.

  5. In the Manual Input view, Use the Stream control to select the InputStream stream.

    Enter any integer into the fib field, such as 90, 500, or 1100.

    Click Send Data.

  6. In the Output Streams view, select the result tuple to see it in the details section below. Look for a tuple field named fibresult that show a list of the members of the Fibonacci series that are less than the integer you sent. Notice that this field is in the top row of three, with the other two fields empty.

  7. Press F9 or click the Stop Running Application button. (We do this to clear the Output Streams view of previous results.)

  8. Double-click the Python adapter icon and open the Script tab. In the Script source control, select File.

    In the Script file control, make sure fib.py is selected. If not, use the Browse button to search the resource path for this file.

  9. Save the module and click the Run button again.

  10. In the Manual Input view, Select the InputStream stream, enter an integer into the fib field, and click Send Data.

  11. In the Output Streams view, look for an output tuple field named fibresultFromFile that shows the Fibonacci series again. Notice that this output fills the middle field this time, with top and bottom fields empty.

  12. Press F9 or click the Stop Running Application button.

  13. Double-click the Python adapter icon again. With either File or Script text selected, make sure Enable control port is selected. (When this checkbox is enabled and the control port is used, it overrides either setting of Script source.)

  14. Save the module and click the Run button again.

  15. To take advantage of the control stream, you must send a replacement script to the running module. To do this, in the Manual Input view, select yControlStream from the Stream control. Then:

    1. In the Command field, enter Load.

    2. In the Script field, copy and paste the following script. Be careful not to change the indent levels. 

      from __future__ import print_function
def fib2(n):
    result = []
    a, b = 0, 1
    while b < n:
        result.append(b)
        a, b = b, a+b
    return result
fibresultFromControlStream = fib2(fib)
print(fibresultFromControlStream)
print('')

      Once pasted, you can only see the top line of the script. Use the up and down arrow keys to scroll through the pasted script in the Script control to make sure the line breaks and indentation pasted correctly.

    3. Click Send Data. There is no feedback for this step.

  16. In the Manual Input view, select the InputStream stream, enter an integer into the fib field, and click Send Data.

  17. In the Output Streams view, look for a tuple this time named fibresultFromControlStream that shows the results of the Fibonacci calculation as before. This time, the output tuple fills the bottom field of the three, with the top and middle fields empty.

  18. When done, press F9 or click the Stop Running Application button.

Sample Location

When you load the sample into StreamBase® Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.

Important

Load this sample in StreamBase® Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.

Using the workspace copy of the sample avoids permission problems. The default workspace location for this sample is: 

studio-workspace/sample_python

See Default Installation Directories for the default location of studio-workspace on your system.