Apache Flume Input and Output Adapters Sample

Contents

About This Sample
Running This Sample in StreamBase Studio
Running This Sample in Terminal Windows
Importing This Sample into StreamBase Studio
Sample Location

This sample demonstrates the use of the TIBCO StreamBase® Input and Output Adapters for Apache Flume. See Apache Flume External Adapters for details on the use and structure of these adapters.

About This Sample

This sample uses one instance of the Flume Output External Adapter to act as a Flume source while connected to the sample StreamBase application. At the same time, a Flume External Input Adapter is running as a Flume sink to receive the Flume events sent by the source and forwards them to the same StreamBase application.

In order to run this sample you must have a machine with the following installed:

StreamBase
The StreamBase Flume adapter, which installs from a separate kit.
Apache Flume version 1.1.0 or later.

Running This Sample in StreamBase Studio

These instructions presume you have installed Flume and have loaded the Flume sample into StreamBase Studio.

Copy $STREAMBASE_HOME/lib/sbflume.jar and $STREAMBASE_HOME/lib/sbclient.jar to flume_installdir/lib as described in Apache Flume External Adapters.
The provided flume.cfg file presumes StreamBase Server runs on the default port, 10000. Edit this file to adjust the port number, if necessary.
In the Package Explorer view, double-click to open the flume.sbapp application. Make sure the application is the currently active tab in the EventFlow Editor.
Click the Run button. This opens the SB Test/Debug perspective and starts the application.
Open a console window and run the following command for Flume 1.1:
```
flume_installdir/bin/flume-ng node host1 -f sb_sample_dir/flume.cfg -c sb_sample_dir/
```
For Flume 1.2, use this command:
```
flume_installdir/bin/flume-ng agent -name host1 -f sb_sample_dir/flume.cfg -c sb_sample_dir/
```
These commands launch the source and sink StreamBase Flume nodes, both of which connect to the configured StreamBase Server and also connect to each other via the configured Flume channel.
Send data through the system by launching the feed simulation. In the SB Test/Debug perspective, select the Feed Simulations view. Look for a simulation named flume-input-data.sbfs in the list of simulations; select it and click the view's Run button.
Now look for tuples sent to the ToFlume output stream, evidenced by the activity in the Application Output view. Each of these tuples is received by the Flume output adapter, which converts it to a Flume event and sends it to the Flume system.
At the same time, look for tuples dequeued from the FromFlume stream, as evidenced by the Application Input view. These are tuples emitted by the Flume Input adapter, converted from the incoming Flume events which were sent by the Output adapter as described above.
To stop the sample, enter Ctrl+C in the console window running the flume-ng script, and press F9 or click the Stop Running Application button in StreamBase Studio.

Running This Sample in Terminal Windows

This section describes how to run the sample in UNIX terminal windows or Windows command prompt windows. On Windows, be sure to use the StreamBase Command Prompt from the Start menu as described in the Test/Debug Guide, not the default command prompt.

Copy $STREAMBASE_HOME/lib/sbflume.jar and $STREAMBASE_HOME/lib/sbclient.jar to flume_installdir/lib as described in Apache Flume External Adapters.
The provided flume.cfg file presumes StreamBase Server runs on the default port, 10000. Edit this file to adjust the port number, if necessary.
Open four terminal windows on UNIX, or four StreamBase Command Prompt on Windows. In each window, navigate to the directory where the sample is installed, or to your workspace copy of the sample, as described above.
In window 1, type:
```
sbd flume.sbapp
```
In window 2, type:
```
sbc dequeue -v
```
This window will display tuples dequeued from the adapter's output ports, both those destined to be sent to Flume and those received from it.
If you are using Flume 1.1, in window 3, type:
```
flume_installdir/bin/flume-ng node host1 -f sb_sample_dir/flume.cfg -c sb_sample_dir/
```
For Flume 1.2, use this command:
```
flume_installdir/bin/flume-ng agent -name host1 -f sb_sample_dir/flume.cfg -c sb_sample_dir/
```
This launches the source and sink StreamBase Flume nodes, both of which connect to StreamBase Server started in window 1, and also connect to each other via the configured Flume channel.
At this point, the system is up and running and only needs some data to run through it. In window 4, enter:
```
sbfeedsim flume-input-data.sbfs
```
This generates data and sends it to the application. In window 2 look for tuples sent to the ToFlume output port, from which the Flume Output adapter dequeues them, converts them to Flume events, and then sends them onto the Flume channel. From there, the Flume Input adapter receives the Flume events, converts them to tuples, and sends them to the application's FromFlume stream. This is evidenced by the console output in window 2.
To stop the sample, enter Ctrl+C in windows 3 and 4. Then, in either of those windows, enter sbadmin shutdown.

Importing This Sample into StreamBase Studio

In StreamBase Studio, import this sample with the following steps:

From the top menu, select File → Load StreamBase Sample.
Select Apache Flume adapter from the StreamBase Messaging Adapters category.
Click OK.

StreamBase Studio creates a single project containing the sample files.

Sample Location

When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.

Important

Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.

Using the workspace copy of the sample avoids the permission problems that can occur when trying to work with the initially installed location of the sample. The default workspace location for this sample is:

studio-workspace/sample_adapter_external_flume

See Default Installation Directories for the location of studio-workspace on your system.

In the default TIBCO StreamBase installation, this sample's files are initially installed in:

streambase-install-dir/sample/adapter/external/flume

See Default Installation Directories for the location of streambase-install-dir on your system. This location may require administrator privileges for write access, depending on your platform.