Contents
This sample demonstrates the use of the TIBCO StreamBase® Input and Output Adapters for Apache Flume. See Apache Flume External Adapters for details on the use and structure of these adapters.
This sample uses one instance of the Flume Output External Adapter to act as a Flume source while connected to the sample StreamBase application. At the same time, a Flume External Input Adapter is running as a Flume sink to receive the Flume events sent by the source and forwards them to the same StreamBase application.
In order to run this sample you must have a machine with the following installed:
-
StreamBase
-
The StreamBase Flume adapter, which installs from a separate kit.
-
Apache Flume version 1.1.0 or later.
These instructions presume you have installed Flume and have loaded the Flume sample into StreamBase Studio.
-
Copy
and$STREAMBASE_HOME
/lib/sbflume.jar
to$STREAMBASE_HOME
/lib/sbclient.jar
as described in Apache Flume External Adapters.flume_installdir
/lib -
The provided
flume.cfg
file presumes StreamBase Server runs on the default port, 10000. Edit this file to adjust the port number, if necessary. -
In the Package Explorer view, double-click to open the
flume.sbapp
application. Make sure the application is the currently active tab in the EventFlow Editor. -
Click the
Run button. This opens the SB Test/Debug perspective and starts the application.
-
Open a console window and run the following command for Flume 1.1:
node host1 -fflume_installdir
/bin/flume-ng
-csb_sample_dir
/flume.cfgsb_sample_dir
/For Flume 1.2, use this command:
agent -name host1 -fflume_installdir
/bin/flume-ng
-csb_sample_dir
/flume.cfgsb_sample_dir
/These commands launch the source and sink StreamBase Flume nodes, both of which connect to the configured StreamBase Server and also connect to each other via the configured Flume channel.
-
Send data through the system by launching the feed simulation. In the SB Test/Debug perspective, select the Feed Simulations view. Look for a simulation named
flume-input-data.sbfs
in the list of simulations; select it and click the view's Run button. -
Now look for tuples sent to the ToFlume output stream, evidenced by the activity in the Application Output view. Each of these tuples is received by the Flume output adapter, which converts it to a Flume event and sends it to the Flume system.
-
At the same time, look for tuples dequeued from the FromFlume stream, as evidenced by the Application Input view. These are tuples emitted by the Flume Input adapter, converted from the incoming Flume events which were sent by the Output adapter as described above.
-
To stop the sample, enter Ctrl+C in the console window running the
flume-ng
script, and press F9 or click thebutton in StreamBase Studio.
This section describes how to run the sample in UNIX terminal windows or Windows command prompt windows. On Windows, be sure to use the StreamBase Command Prompt from the Start menu as described in the Test/Debug Guide, not the default command prompt.
-
Copy
and$STREAMBASE_HOME
/lib/sbflume.jar
to$STREAMBASE_HOME
/lib/sbclient.jar
as described in Apache Flume External Adapters.flume_installdir
/lib -
The provided
flume.cfg
file presumes StreamBase Server runs on the default port, 10000. Edit this file to adjust the port number, if necessary. -
Open four terminal windows on UNIX, or four StreamBase Command Prompt on Windows. In each window, navigate to the directory where the sample is installed, or to your workspace copy of the sample, as described above.
-
In window 1, type:
sbd flume.sbapp
-
In window 2, type:
sbc dequeue -v
This window will display tuples dequeued from the adapter's output ports, both those destined to be sent to Flume and those received from it.
-
If you are using Flume 1.1, in window 3, type:
node host1 -fflume_installdir
/bin/flume-ng
-csb_sample_dir
/flume.cfgsb_sample_dir
/For Flume 1.2, use this command:
agent -name host1 -fflume_installdir
/bin/flume-ng
-csb_sample_dir
/flume.cfgsb_sample_dir
/This launches the source and sink StreamBase Flume nodes, both of which connect to StreamBase Server started in window 1, and also connect to each other via the configured Flume channel.
-
At this point, the system is up and running and only needs some data to run through it. In window 4, enter:
sbfeedsim
flume-input-data.sbfs
This generates data and sends it to the application. In window 2 look for tuples sent to the
ToFlume
output port, from which the Flume Output adapter dequeues them, converts them to Flume events, and then sends them onto the Flume channel. From there, the Flume Input adapter receives the Flume events, converts them to tuples, and sends them to the application'sFromFlume
stream. This is evidenced by the console output in window 2. -
To stop the sample, enter Ctrl+C in windows 3 and 4. Then, in either of those windows, enter
sbadmin shutdown
.
In StreamBase Studio, import this sample with the following steps:
-
From the top menu, select
→ . -
Select
Apache Flume adapter
from the StreamBase Messaging Adapters category. -
Click OK.
StreamBase Studio creates a single project containing the sample files.
When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.
Important
Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.
Using the workspace copy of the sample avoids the permission problems that can occur when trying to work with the initially installed location of the sample. The default workspace location for this sample is:
studio-workspace
/sample_adapter_external_flume
See Default Installation Directories for the location of studio-workspace
on your system.
In the default TIBCO StreamBase installation, this sample's files are initially installed in:
streambase-install-dir
/sample/adapter/external/flume
See Default Installation Directories for the default location of studio-workspace
on your system.