Contents
These samples illustrate how to access the Amazon S3 file system using the TIBCO StreamBase® File Writer for Apache Hadoop Distributed File System (HDFS) and TIBCO StreamBase® File Reader for Apache Hadoop Distributed File System (HDFS).
In StreamBase Studio, import this sample with the following steps:
-
From the top-level menu, select
> . -
Enter
s3
to narrow the list of options. -
Select S3 file system access via HDFS from the Large Data Storage and Analysis category.
-
Click
.
StreamBase Studio creates a single project containing the sample files.
Open the sample application, FileBasic.sbapp
or FileAdvanced.sbapp
, select the Parameters tab, and edit the value to represent:
-
Your current S3 bucket
-
Where you would like to store the sample data
You must also open the sample S3 configuration file, sample_s3a.xml
, and enter your security access keys for your S3 file system. See this link for other ways to authenticate with S3: Authenticating with S3.
-
In the Project Explorer view, open this sample's folder.
Keep an eye on the bottom right status bar of the Studio window. Make sure any
Updating
,Downloading
,Building
, orRebuild project
messages finish before you proceed. -
Open the
src/main/eventflow/
folder.packageName
-
Double-click to open the
FileBasic.sbapp
module. Make sure the module is the currently active tab in the EventFlow Editor. -
Click the
Run button. This opens the SB Test/Debug perspective and starts the module.
-
Wait for the Waiting for fragment to initialize message to clear.
-
In the Manual Input view, switch the
Stream
toData
, then enter a string value such astest
. -
Click
to send a data tuple to be written to the file. Repeat for as many lines as you wish. -
In the Output Streams view, observe tuples emitted on the
Status
output streams indicating actions performed on the file. -
In the Manual Input view, switch the
Stream
toWriteControl
, then enterClose
into theCommand
field. -
Click
to send a control tuple, which closes the current file for writing. -
In the Manual Input view, switch the
Stream
toReadControl
, then click to send a control tuple, which reads the default file. -
Press F9 or click the
Terminate EventFlow Fragment button.
-
This demo has now created a file in your S3 file system called
sample/Sample.txt
containing the lines of data you submitted.
-
In the Project Explorer view, open this sample's folder.
Keep an eye on the bottom right status bar of the Studio window. Make sure any
Updating
,Downloading
,Building
, orRebuild project
messages finish before you proceed. -
Open the
src/main/eventflow/
folder.packageName
-
Double-click to open the
FileAdvanced.sbapp
module. Make sure the module is the currently active tab in the EventFlow Editor. -
Click the
Run button. This opens the SB Test/Debug perspective and starts the module.
-
Wait for the Waiting for fragment to initialize message to clear.
-
In the Output Streams view, observe tuples emitted on the
Status
output streams indicating actions performed to the files. -
Press F9 or click the
Terminate EventFlow Fragment button.
-
This demo has now created multiple files in your S3 file system:
-
sample/Sample.gz
— This file is a GZip compressed file created from theSampleIn.txt
file. -
sample/Sample.gz2
— This file is a BZip2 compressed file created from theSampleIn.txt
file. -
sample/Sample.zip
— This file is a Zip compressed file created from theSampleIn.txt
file. -
sample/SampleOut.txt
— This file is an uncompressed file created from theSampleIn.txt
file.
-
When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.
Important
Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.
Using the workspace copy of the sample avoids permission problems. The default workspace location for this sample is:
studio-workspace
/sample_adapter_embedded_hdfsS3
See Default Installation Directories for the default location of studio-workspace
on your system.