Apache Flume Input and Output Adapters Sample

Copy $STREAMBASE_HOME/lib/sbflume.jar and $STREAMBASE_HOME/lib/sbclient.jar to flume_installdir/lib as described in Apache Flume External Adapters.

The provided flume.cfg file presumes StreamBase Server runs on the default port, 10000. Edit this file to adjust the port number, if necessary.

In the Package Explorer view, double-click to open the flume.sbapp application. Make sure the application is the currently active tab in the EventFlow Editor.

Click the Run button. This opens the SB Test/Debug perspective and starts the application.

Open a console window and run the following command for Flume 1.1:

flume_installdir/bin/flume-ng node host1 -f sb_sample_dir/flume.cfg -c sb_sample_dir/

For Flume 1.2, use this command:

flume_installdir/bin/flume-ng agent -name host1 -f sb_sample_dir/flume.cfg -c sb_sample_dir/

These commands launch the source and sink StreamBase Flume nodes, both of which connect to the configured StreamBase Server and also connect to each other via the configured Flume channel.

Send data through the system by launching the feed simulation. In the SB Test/Debug perspective, select the Feed Simulations view. Look for a simulation named flume-input-data.sbfs in the list of simulations; select it and click the view's Run button.

Now look for tuples sent to the ToFlume output stream, evidenced by the activity in the Application Output view. Each of these tuples is received by the Flume output adapter, which converts it to a Flume event and sends it to the Flume system.

At the same time, look for tuples dequeued from the FromFlume stream, as evidenced by the Application Input view. These are tuples emitted by the Flume Input adapter, converted from the incoming Flume events which were sent by the Output adapter as described above.

To stop the sample, enter Ctrl+C in the console window running the flume-ng script, and press F9 or click the Stop Running Application button in StreamBase Studio.

Copy $STREAMBASE_HOME/lib/sbflume.jar and $STREAMBASE_HOME/lib/sbclient.jar to flume_installdir/lib as described in Apache Flume External Adapters.

The provided flume.cfg file presumes StreamBase Server runs on the default port, 10000. Edit this file to adjust the port number, if necessary.

Open four terminal windows on UNIX, or four StreamBase Command Prompt on Windows. In each window, navigate to the directory where the sample is installed, or to your workspace copy of the sample, as described above.

In window 1, type:

sbd flume.sbapp

In window 2, type:

sbc dequeue -v

This window will display tuples dequeued from the adapter's output ports, both those destined to be sent to Flume and those received from it.

If you are using Flume 1.1, in window 3, type:

flume_installdir/bin/flume-ng node host1 -f sb_sample_dir/flume.cfg -c sb_sample_dir/

For Flume 1.2, use this command:

flume_installdir/bin/flume-ng agent -name host1 -f sb_sample_dir/flume.cfg -c sb_sample_dir/

This launches the source and sink StreamBase Flume nodes, both of which connect to StreamBase Server started in window 1, and also connect to each other via the configured Flume channel.

At this point, the system is up and running and only needs some data to run through it. In window 4, enter:

sbfeedsim flume-input-data.sbfs

This generates data and sends it to the application. In window 2 look for tuples sent to the ToFlume output port, from which the Flume Output adapter dequeues them, converts them to Flume events, and then sends them onto the Flume channel. From there, the Flume Input adapter receives the Flume events, converts them to tuples, and sends them to the application's FromFlume stream. This is evidenced by the console output in window 2.

To stop the sample, enter Ctrl+C in windows 3 and 4. Then, in either of those windows, enter sbadmin shutdown.