HDFS Binary File Reader/Writer Adapter Sample

This sample demonstrates the use of the TIBCO StreamBase® Binary File Reader and Writer Adapters for Apache Hadoop Distributed File System (HDFS).

Initial Setup

You must open the sample application, binaryrw.sbapp and select the Parameters tab and edit the HDFS_FILE_PATH and HDFS_USER to represent your current HDFS setup and where you would like to store the sample data.

Running This Sample in StreamBase Studio

  1. In the Package Explorer view, double-click myfile.csv to see the CSV records that will be converted to binary.

  2. Double-click to open the sample application, binaryrw.sbapp.

  3. Click the Run button. This opens the Test/Debug perspective and starts the application.

  4. In the Manual Input view, select the DoWrite input stream and press Send Data.

  5. Look for tuples emitted on the BinaryDataWritten output stream and observe they contain the values from myfile.csv.

  6. Select the DoRead input stream and press Send Data.

  7. Look for tuples emitted on the BinaryDataRead output stream and observe they contain the values from myfile.csv.

  8. Select the DoDelete input stream and press Send Data.

  9. Look for tuples emitted on the DeleteStatus output stream to confirm the file has been deleted.

  10. When done, press F9 or click the Stop Running Application button.

Running This Sample in Terminal Windows

This section describes how to run the sample in UNIX terminal windows or Windows command prompt windows. On Windows, be sure to use the StreamBase Command Prompt from the Start menu as described in the Test/Debug Guide, not the default command prompt.

  1. Open three terminal windows on UNIX, or three StreamBase Command Prompt s on Windows. In each window, navigate to your workspace copy of the sample, as described above.

  2. In window 1, type:

    sbd binaryrw.sbapp

  3. In window 2, type:

    sbc dequeue -v

    This window will display the tuples dequeued from the adapter's output port.

  4. In window 3, type the following command to cause the sample application to write the CSV file contents to a binary file:

    echo null | sbc enqueue DoWrite

  5. Observe in window 2 that tuples are emitted from the BinaryDataWritten stream.

  6. In window 3, type the following command to cause the sample application to read the read and display the contents of the binary file:

    echo null | sbc enqueue DoRead

  7. Observe in window 2 that tuples are emitted from the BinaryDataRead stream.

  8. In window 3, type the following command to cause the sample application to delete the binary file:

    echo null | sbc enqueue DoDelete

  9. Observe in window 2 that tuples are emitted from the DeleteStatus stream.

  10. In window 3, type the following command to terminate the server and dequeuer:

    sbadmin shutdown

Importing This Sample into StreamBase Studio

In StreamBase Studio, import this sample with the following steps:

  • From the top menu, click FileLoad StreamBase Sample.

  • Select this sample from the Embedded Input Adapters list.

  • Click OK.

StreamBase Studio creates a project for this sample.

Sample Location

When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.

Important

Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.

Using the workspace copy of the sample avoids the permission problems that can occur when trying to work with the initially installed location of the sample. The default workspace location for this sample is:

studio-workspace/sample_hdfsbinaryrw

See Default Installation Directories for the location of studio-workspace on your system.

In the default TIBCO StreamBase installation, this sample's files are initially installed in:

streambase-install-dir/sample/hdfsbinaryrw

See Default Installation Directories for the location of streambase-install-dir on your system. This location may require administrator privileges for write access, depending on your platform.