< Previous		Next >

Twitter Input Adapter

Contents

Introduction
Stream Types
Adapter Properties
Control Port
Obtaining Twitter OAuth Tokens
Related Topics

Introduction

The Spotfire Streaming Twitter Input adapter connects a StreamBase application to the web service provided by Twitter, Inc. It can receive Twitter status messages and direct messages, commonly called tweets.

The input adapter can be initially set up to respond to control messages for changing at runtime the stream being watched. It can watch the user, sample, link, and filter twitter streams, and can perform searches. The adapter can also process direct messages on the user stream, if so enabled.

By default, the input adapter is configured to not connect to any twitter streams on startup, awaiting a control message to perform an operation.

The adapter must be configured with the correct OAuth tokens for it to perform any operations. The OAuth can be set up in the security tab, as described in Obtaining Twitter OAuth Tokens below.

The adapter has samples, described in Twitter Adapter Sample.

Fields in schemas used by the Twitter adapters correspond to attributes in JSON-encoded tweets accessed via Twitter4J library calls. The library is provided with StreamBase. To help you understand what fields in schemas represent, the following table summarizes Twitter objects and attributes, and provides links to their documentation at dev.twitter.com.

Object	Example Attributes	Reference
Tweet	id, following, followers_count, geo_enabled, status	Platform Objects:Tweets
User	id, following, followers_count, geo_enabled, status	Platform Objects:Users
Geo Place	country_code, locality, region, street_address, postal_code	Platform Objects:Places
Entity	hashtags, media, urls, user_mentions	Platform Objects:Entities

Briefly,

Tweets are the basic atomic building block of all things Twitter. Users tweet Tweets, also known more generically as status updates. Tweets can be embedded, replied to, favorited, unfavorited and deleted.
Users can be anyone or anything. They tweet, follow, create lists, have a home_timeline, can be mentioned, and can be looked up in bulk.
Places are specific, named locations with corresponding geo coordinates. They can be attached to Tweets by specifying a place_id when tweeting.
Entities provide metadata and contextual information about content posted on Twitter and are not autonomous objects.

Schemas for the Spotfire Streaming Twitter adapters encode a large subset of attributes for these objects. You can modify the schemas to include others.

Stream Types

The input adapter has multiple twitter stream types it can handle. Each separate input adapter can handle one of the following at a time:

Filter—Start consuming public statuses that match one or more keywords or user IDs. At least one keyword or user ID must be specified. Multiple parameters may be specified, allowing most clients to use a single connection to the Streaming API. The default access level allows tracking up to 200 keywords, 400 follow user IDs, and ten 1-degree location boxes. Increased access levels allow 80,000 follow user IDs ("shadow" role), 400,000 follow user IDs ("birddog" role), 10,000 track keywords ("restricted track" role), 200,000 track keywords ("partner track" role), and 200 10-degree location boxes ("locRestricted" role). Increased track access levels also pass a higher proportion of statuses before limiting the stream.

Note

To increase your access level (to gardenhose or firehose levels), you must make arrangements with Twitter, Inc. and possibly with one of its resellers. Contact the Twitter Certified Products Program, and also consult its Developer Rules of the Road page.
Sample—Starts listening to a random sample of all public statuses. The default access level provides a small proportion (1% or less) of the Firehose. The gardenhose access level provides a proportion more suitable for data mining and research applications that desire a larger proportion to make up a statistically significant sample.
Link—Starts listening on the link feed, which publishes all status updates that contain Web links.
**(Deprecated)User—User Streams provide a stream of data and events specific to the authenticated user and provide real-time updates of all data needed to update a desktop application display of the current user's feed, with almost no rate limits or latency. This feed also produces direct messages.

Adapter Properties

This section describes the properties you can set for this adapter, using the various tabs of the Properties view in StreamBase Studio.

General Tab

Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.

Adapter: A read-only field that shows the formal name of the adapter.

Class name: Shows the fully qualified class name that implements the functionality of this adapter. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.

Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.

Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.

Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

Adapter Properties Tab

Property	Description
Enable Direct Messages Port	Enabling this port allows your application to receive tweets directly from users. Your API key must be have appropriate security to allow this feature.
Enable Users Port	The users output port is used for the SearchUsers, Followers, and Following commands.
Enable Places Port	The places output port is used for SearchPlaces commands.
Enable Friendships Port	The friendships output port is used for LookupFriendship commands.
Enable Trends Port	The trends output port is used for SearchTrends commands.
Enable Rate Limits Status Port	The rate limits status output port is used for RateLimitsStatus commands.
Enable Status Port	The status output port emits status messages for your application's consumption.
Startup Twitter Stream	The initial stream to connect to when the application is started.
Keywords	The keywords to use when the start up type is set up to filter.
User Ids	The user IDs to use when the start up type is set up to filter.
Log Level	Controls the level of verbosity the adapter uses to send notifications to the console. This setting can be higher than the containing application's log level. If set lower, the system log level is used. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE.

Security Tab

Property	Type	Description
Twitter Consumer Key	string	Obtain Consumer Key from app @ developer.twitter.com
Twitter Consumer Secret	string	Obtain Consumer Secret from app @ developer.twitter.com
Twitter Access Token Secret	string	Obtain Access Token Secret from app @ developer.twitter.com
Twitter Access Token	string	Obtain Access Token from app @ developer.twitter.com

Cluster Aware Tab

Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.

Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.

Concurrency Tab

Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.

Caution

Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.

Control Port

Description

The control port is used to send action commands to the adapter.

Control Port Schema

Command, string. The command to send to the adapter. Specify one of the following:
- Filter—This command tells the adapter to listen to a filter stream based on the criteria given in the Filter tuple.
- Sample—This command tells the adapter to start listening to the sample twitter stream.
- Link—This command tells the adapter to start listening to the link twitter stream. This stream needs special permission from Twitter, Inc. to use.
- **(Deprecated)User—This command tells the adapter to start listening to the current users twitter stream and to direct messages, if enabled.
- SearchTweets—This command performs a search of all tweets based on the criteria given in the SearchTweet tuple.
- SearchUsers—This command performs a search of users based on the criteria given in the SearchUsers tuple.
- SearchPlaces—This command performs a search of place based on the criteria given in the SearchPlaces tuple.
- SearchTrends—This command performs a search for trends in a geographic location using the SearchTrends tuple.
- Timeline—This command tries to receive as many historical status updates from the current user's own timeline as it can.
- LookupFriendship—This command looks up the friendship relationship between the current user and the specified ScreenName.
- Follow—Start following a particular user.
- Unfollow—Stop following a particular user.
- Followers—List the current user's followers.
- Following—List the users that the current user is following.
- RateLimitsStatus—Outputs all the current user's rate limits to determine whether we can perform more operations.
- Disconnect—tells the adapter to disconnect from the current twitter stream.
Filter, Tuple. The tuple used to create the filter stream. This field is only used when the Command is filter, and is set to one of:
- Count, int. The number of tweets to go back to in history and fetch, before starting to stream.
- UserIDs, list(long), (optional). A list of user IDs to add to this stream.
- Keywords, list(string), (optional). A list of keywords to use when creating this stream.
SearchTweets, Tuple, the tuple used to search. This field is only used when the Command is SearchTweets.
- SearchTerm, string. The text to search tweets for.
- Count, int. The maximum number of results to return. Please note that Twitter has its own maximum that may be less than this number.
- Since, timestamp. The starting date to look for tweets.
- GeoLocation, Tuple. The geo location to search for tweets.
  - Longitude, double. The longitude point to search.
  - Latitude, double. The latitude point to search.
  - Radius, double. The radius search starting from the position specified by the longitude and latitude.
  - Units, string. The units to use for the radius: km for kilometer, mi for miles.
SearchUsers, Tuple. The tuple used to search for users. This field is only used when Command is SearchUsers.
- SearchTerm, string. The text to search tweets for.
SearchPlaces, Tuple, the tuple used to search for locations. This field is only used when Command is SearchPlaces.

Either IP or Longitude and Latitude can be specified but only one will be used.
- Longitude, double. The latitude to search near. This parameter is ignored unless it is inside the range -90.0 to +90.0 (North is positive), inclusive. It is also ignored if there is no corresponding Latitude parameter.
- Latitude, double, The longitude to search near. The valid range for longitude is -180.0 to +180.0 (East is positive), inclusive. This parameter is ignored if outside that range, if it is not a number, or if there is no corresponding Longitude parameter.
- IP, string. An IP address used when attempting to assign geolocation based on the user's IP address. Example Values: 74.125.19.104
- Granularity, string. This is the smallest region to return, and must be one of:
  - poi
  - neighborhood
  - city
  - admin
  - country
  If no granularity is specified for the request, neighborhood is assumed. Specifying a granularity implies accepting all larger geographical units. For example, if set to city, results include places of type city, admin, and country.
- Accuracy, string. A hint on the region in which to search. If a number, then this specifies a radius in meters. It can also be a string that must be a number suffixed with ft to specify feet, such as 100ft. If Accuracy is not specified, it defaults to 0m. In practice, accuracy is whatever accuracy the user's device is capable of measuring at its current location.
ScreenName, String. The screen name to use with the Command Follow or Unfollow.
SearchTrends, Tuple. The tuple used to search for trends. This field is only used when Command is SearchTrends.
- GeoLocation (Both fields must have valid values.)
  - Longitude, double. The latitude to search near. The valid range is -90.0 to +90.0 (North is positive), inclusive.
  - Latitude, double. The longitude to search near. The valid range is -180.0 to +180.0 (East is positive), inclusive.

Obtaining Twitter OAuth Tokens

The Twitter adapter must be configured with valid Twitter OAuth consumer and access tokens, obtained without charge from Twitter's developer web site. Twitter OAuth tokens encapsulate the login credentials for one Twitter account. The tokens encapsulate your Twitter account's authentication rights without storing or requiring your account's password, and thereby remain effective even when you change your Twitter password. For additional details, see the Twitter OAuth documentation.

Follow the steps below to obtain OAuth tokens from Twitter.

Decide in advance which Twitter account you will use for communication with your StreamBase application. You probably do not want to use a personal account for business communication, so set up a new business Twitter account if one does not already exist.
Proceed to http://dev.twitter.com and use the Sign in link to log in with the designated Twitter account.
Your Twitter account name is now displayed where the sign-in link was. Hover over it to display a context menu and select My Applications.
Click the Create a new application link. This opens a form titled Create an Application.
- Fill in a Name. This name must be unique for the logged-in Twitter account.
- Fill in the Description field. Twitter requires at least 10 characters in this field.
- Provide a URL in the Website field.
- For this example, leave the Callback URL blank.
- Accept the Develop Rules Of The Road by checking the Yes, I agree box.
- Provide the CAPTCHA response.
- Click the Create your Twitter application button.
The application has been created. You will see the screen shown below. Click the Settings tab.
Under the Application Type section select the Read and Write radio button (Note: if you plan to send and receive direct messages with your application, you must instead select Read, Write and Access direct messages). This allows your application to receive and post tweets. Click Update this Twitter application's settings to complete the update.
Click the Details tab.
At the bottom of the page click Create my access token tab.
Copy, label, and save the Consumer Key , Consumer Secret , Access Token and Access Token Secret strings provided on this page. You will need to enter these tokens in your Twitter input adapter's StreamBase Properties>Security tab.