ETP Adapters

< Previous		Next >

Introduction

The Spotfire Streaming Adapter for ETP allows StreamBase to interact with data from an ETP (Energistics Transfer Protocol) v1.1 server. The adapters in this set include:

The ETP Control adapter, which is used to control the connection with the ETP server (connection, disconnection, server status, and so on).
The ETP Store Get adapter, which is used to retrieve values from ETP nodes.
The ETP Discovery adapter, which is used to traverse the ETP server's node hierarchy and retrieve metadata for these nodes.
The ETP Streaming adapter, which is used to subscribe to value nodes and get continuous updates when the nodes' values change, as well as perform range requests for such values.

Placing an ETP Adapter on the Canvas

The ETP adapters are members of the Java Operators group in the Palette view in StreamBase Studio. Select the operator from the Insert an Operator or Adapter dialog, following these steps:

Open the Insert an Operator or Adapter dialog with one of the following methods:
1. Drag the Adapters, Java Operators token from the Operators and Adapters drawer of the Palette view to the canvas.
2. Click on the canvas where you want to place the operator, then invoke the keyboard shortcut O V.
3. From the top-level menu, invoke Insert>Operator>Java.
In the search field, enter ETP to narrow the list of adapters.
Select the ETP adapter of interest and click OK.

Prerequisites

In order run correctly, the adapters require that the Microsoft .NET Framework 4.8 or higher be installed on the machine.

How the Adapters Work

All adapters have one input stream to receive commands. The schema for this command stream varies depending on the adapter type; each schema is described in a section below.

All adapters have an output stream to communicate status events (such as when an error occurs during the execution of a command). The schema for this stream is identical for all ETP adapter types (save for its inputTuple tuple field, whose schema will vary according to the operator's input stream's schema) and is described in a section below.

With the exception of the Control adapter, all adapters also have a second output port to emit the results of running commands. For example, the Store Get adapter uses this port to emit the value(s) it reads from the server.

Finally, the Streaming adapter has an additional port to emit metadata collected for requested nodes.

Configuration

The different ETP adapters share a connection to the same server, provided they are configured to do so. Each adapter lists the servers available in a combo box (see ETP Adapter Properties). The list's values are specified in a dedicated section of the application's adapter configuration file. Here is an example of such a section, containing all supported settings (long lines wrap to the next, for clarity):

name = "ETP.conf"
type = "com.tibco.ep.streambase.configuration.adapter"
version = "1.0.0"
configuration = {

  // An adapter group type defines a collection of EventFlow adapter configurations, 
  // indexed by adapter type.
  AdapterGroup = {

      // A collection of EventFlow adapter configurations, indexed by adapter type. 
      // This object is required and // must contain at least one configuration.
      adapters = {

        // The root section for an EventFlow adapter configuration.
        etp = {

          // Section list. This array is optional and has no default value.
          sections = [ 

            // A configuration for an EventFlow adapter named section.
            {

            // Section name. The value does not have to be unique; that is, you can have 
            // multiple sections with the same name in the same array of sections. This property
            // is required.
            name = "server-definition"

            // Section of properties. All values must be strings. This object is optional 
            // and has no default value.
            settings = {
              id = "MyETPServer"
                            
              // The ETP server to which to connect
              server-url = "ws://localhost/Witsml.Web/api/etp"
                            
              // 0 to not re-attempt connection when the attempt fails, -1 to continuously re-attempt
              num-reconnection-retries = "5"
              reconnection-retry-interval-ms = "10000"

              // Credentials to use for connection (leave empty for anonymous connections)
              // Note that you can encipher "password" using the "epadmin encrypt secret" command
              username = ""
              password = ""
                            
              // Proxy-related setting. Leave empty when not using a proxy to connect to ETP server
              // Note that you can encipher "proxy-password" using the "epadmin encrypt secret" command
              proxy-host = ""
              proxy-port = ""
              proxy-username = ""
              proxy-password = ""
           }
         }
       ]
      }
    }
  }
}

A best practice is to define your endpoints in this file before placing operator instances on the canvas, so that the lists are already available in the Properties view and the operators can be configured right away.

ETP Adapter Properties

This section describes the properties you can set for the adapters, using the various tabs of the Properties view in StreamBase Studio.

In following section tables, the Property column shows each property name as found in the one or more adapter properties tabs of the Properties view for this adapter.

Use the StreamSQL names of the adapter's properties when using this adapter in a StreamSQL program with the APPLY JAVA statement.

ETP General Tab

Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.

Operator: A read-only field that shows the formal name of the operator.

Class name: Shows the fully qualified class name that implements the functionality of this operator. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.

Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this operator starts.

Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.

Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

ETP Operator Properties

The ETP adapters have the following properties, with the exceptions noted for some properties only on particular adapters:

Property	Data Type	Default	Description
ETP Server Definition	combo box	Cleared	REQUIRED. Specifies the ID of the ETP server to which to connect, as defined in the adapter configuration file.
Connect on Startup	check box	Cleared	CONTROL ADAPTER ONLY. Attempt to connect to the server when the application starts.
Log Level	drop-down list	INFO	Controls the level of verbosity the adapter uses to send notifications to the console. This setting can be higher than the containing application's log level. If set lower, the system log level will be used. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE.

Cluster Aware Tab

Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.

Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.

Concurrency Tab

Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.

Caution

Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.

Input Ports

The adapters have a single input port which is used to receive commands. The schema for this port is different for each adapter; those schemas are described in this section.

ETP Control Adapter Input Port

This port is used to connect or disconnect from the configured server, and to get the current connection status. Results are issued on the adapter's Status output port.

Field Name	Field Type	Description
command	string	REQUIRED. The command to execute. One of connect, disconnect, getconnectionstatus. Values are case-insensitive.

ETP Discovery Adapter Input Port

The Discovery adapter allows your application to traverse the data hierarchy of the ETP server. Using this port you can retrieve the metadata of the child nodes of the node specified in the uri field. This metadata includes the child node's uri, content type and several other items. For any given query, one tuple is emitted on the Discovery adapter's metadata output port (see the description of this output port below) for every child node found.

Field Name	Field Type	Description
uri	string	REQUIRED. Contains the URI of the node for which to retrieve child node metadata. If no value is provided, the root node of the server will be queried.

ETP Store Get Adapter Input Port

This port instructs the adapter to retrieve the current value of the specified ETP node. The retrieved data will be emitted on the results output port of the adapter (see the description of this output port).

Field Name	Field Type	Description
uri	string	REQUIRED. Contains the URI of the node from which to retrieve data.

ETP Streaming Adapter Input Port

This port is used to start (and stop) monitoring one or more data nodes for changes in their value. Periodical updates are issued by the adapter for each of these subscriptions until the subscription is removed via a StreamingStop command to this port. Multiple subscriptions may be active at the same time.

Before a streaming session can be initiated, a Start command must be sent to the server via the adapter's input port, optionally specifying values for the maxDataItems and maxMessageRate fields.

ETP servers come in two configurations: Simple and Non-Simple. Simple streaming servers start sending channel updates for all values immediately after the Start command is received. For Non-Simple servers, a Describe command must also be sent, followed by the StreamingStart command to specify exactly which data points to stream.

Finally, you can use this port to issue ETP range requests, via the RangeRequest command.

Field Name		Field Type	Description
command		string	REQUIRED. One of Start, Describe, StreamingStart, StreamingStop, RangeRequest
maxDataItems		int	Start command only—OPTIONAL. The maximum number of items to return in an update.
maxMessageRate		int	Start command only—OPTIONAL. The rate at which updates are emitted. That is, the minimum amount of time (in milliseconds) between updates.
uris		list<string>	Describe command only—REQUIRED. The URIs of the nodes to describe.
channelInfos		list<tuple>	StartStreaming command only—REQUIRED. The information on each channel to start streaming.
^	channelID	long	The ID of the channel to stream.
	startIndex	double	The index at which to start streaming.
	receiveNotifications	boolean	Whether to receive updates (`true`) or just a single snapshot (`false`).
channelIDs		list<long>	StopStreaming command only—REQUIRED. The IDs of the nodes to stop streaming.
rangeInfos		list<tuple>	RangeRequest command only—REQUIRED. The information for each node for which to request the range.
^	channelIDs	list<long>	The IDs of the nodes to request.
	startIndex	long	The index at which to start.
	endIndex	long	The index at which to stop.

Output Ports

Every adapter (except the Control adapter) has an output port to emit tuples resulting from running queries. In addition, every adapter has an output port to receive status events. Finally, the Streaming adapter also has a third port to emit metadata tuple.

Status Output Port (Common to all ETP operators)

The status port emits tuples that describe the status of processing each input tuple. The schema of the output tuple is as follows:

Field Name		Field Type	Description
status		string	The type of status represented by this tuple (`Connected`, `Disconnected`, `Success, Error`, and so on).
message		string	A description of the event.
exception		string	If this event was generated as a result of an exception, this field describes the exception. Otherwise, it is left null.
time		timestamp	The time at which this status tuple was generated.
serverInfo		tuple	If applicable, this tuple will contain more information on the server related to this status event. When this tuple is set, its serverID will always be non-null but all other values may be left empty depending on what information is available on it at the time the status tuple is generated.
^	serverID	string	The ID of the server, as it appears in the adapter's configuration file (see Configuration above).
	serverURL	string	The server's URL.
	sessionID	string	The ID of the current session with the server.
	applilcationName	string	The server's application name
	applicationVersion	string	The server version.
	connectionStatus	string	Typically used in reply to the Control adapter's GetConnectIonStatus command. One of `Connected`, `Disconnected`.
	supportedProtocols	list<string>	A list of the ETP protocols that this server supports. Typically set for a `Connected` event.
	supportedObjects	list<string>	A list of the ETP data types (expressed as MIME strings) that this server supports. Typically set for a `Connected` event.
inputTuple		tuple	The input tuple that caused this status tuple to be emitted, if any.

ETP Discovery Adapter Results Output Port

The Discovery adapter allows your application to traverse the data hierarchy of the ETP server. This port emits tuples describing the metadata of a node on the server. This metadata includes the node's name, uri and content type, whether it has child nodes and so forth. One tuple will be emitted on this port for every child node found at the uri provided in the command input tuple.

Field Name			Field Type	Description
header			tuple	The header of the ETP message represented by this tuple.
^	protocol		int
	messageType		int
	correlationId		long
	messageId		long
	messageFlags		int
resource			tuple	The description of this node.
^	uri		string	The uri of this resource.
	contentType		string	The content type (typically in MIME format) of this resource.
	name		string	The name of this resource.
	channelSubscribable		bool	Indicates that this resource is the source of one or more channels for streaming. Specifically, the URI in this record can be sent in a Describe command in the Streaming adapter with an expectation of receiving one or more channels.
	resourceType		string	The type of this resource.
	hasChildren		int	Indicated the number of children for this resource: -1: Child count is unknown 0: No Children Positive value: Child count
	uuid		string
	lastChanged		int
	objectNotifiable		boolean	Whether this node is streamable.
	customData		list<tuple>	The list of key-value pairs representing any additional data for this resource.
	^	key	string
	^	value	string
Input			tuple	The command tuple sent to the input port that triggered this result.

ETP Get Adapter Results Output Port

The Get adapter allows your application to query the current value of a node on the ETP server. All retrieved values are given as strings unless such a conversion is not possible, in which case the value will be given as a blob.

Field Name				Field Type	Description
header				tuple	The header of the ETP message represented by this tuple.
^	protocol			int
	messageType			int
	correlationId			long
	messageId			long
	messageFlags			int
dataObject				tuple	The data retrieved for this node.
^	resource			tuple	The value of this node.
	^	uri		string	The uri of this resource.
		contentType		string	The content type (typically in MIME format) of this resource.
		name		string	The name of this resource.
		channelSubscribable		bool	Indicates that this resource is the source of one or more channels for streaming. Specifically, the URI in this record can be sent in a Describe command in the Streaming adapter with an expectation of receiving one or more channels.
		resourceType		string	The type of this resource.
		hasChildren		int	Indicates the number of children for this resource: -1: Child count is unknown 0: No Children Positive value: Child count
		uuid		string
		lastChanged		int	Time
		objectNotifiable		boolean	Whether this node is streamable.
		customData		list<tuple>	The list of key-value pairs representing any additional data for this resource.
		^	key	string
		^	value	string
contentEncoding				string	The data encoding scheme used by this node
dataAsBytes				blob	The data retrieved, as a blob. This is null unless `dataAsString` is null.
dataAsString				string	The data retrieved, as a string. If the data cannot be represented as a string, this field is left null and `dataAsBytes` is set instead.

ETP Streaming Adapter Output Ports

Use the Streaming adapter to retrieve metadata and data values for a given list of ETP nodes. The Describe command of this operator (see the description of this input port) will issue the requested metadata to the Metadata Results output port, while the StartStreaming and RangeRequest commands will output data to the Data Results output port.

ETP Streaming Adapter Metadata Results Output Port

Describe commands set to this operator will issue tuples to this port.

Field Name					Field Type	Description
header					tuple	The header of the ETP message represented by this tuple.
^	protocol				int
	messageType				int
	correlationId				long
	messageId				long
	messageFlags				int
channels					list<tuple>	The list of metadata records retrieved.
^	channelID				string	The ID of the channel represented by this metadata record.
	channelName				string
	channelURI				string
	dataType				string
	uom				string	Unit of measure.
	startIndex				long
	stopIndex				long
	description				string
	status				string
	contentType				string
	source				string
	measureClass				string
	uuid				string
	domainObject				tuple
	^	resource			tuple
		^	uri		string
			contentType		string
			name		string
			channelSubscribable		boolean	Whether this node is streamable.
			resourceType		string
			hasChildren		int	Indicates the number of children for this resource: -1: Child count is unknown 0: No Children Positive value: Child count
			uuid		string
			lastChanged		long
			objectNotifiable		boolean
			customData		list<tuple>
			^	name	string
			^	value	string
		contentEncoding			string
		dataAsBytes			blob
		dataAsString			string
	indexes				list<tuple>
	^	uri			string
		indexType			string
		uom			string
		depthDatum			string
		direction			string
		mnemonic			string
		description			string
		scale			string
		timeDatum			string
		customData			string
		^	name		string
		^	value		string
	customData				list<tuple>
	^	name			string
	^	value			string

ETP Streaming Adapter Data Results Output Port

StartStreaming and RangeRequest commands sent to this adapter will output their data to this port. The Streaming adapter allows your application to retrieve the data for one or more streamable nodes. Additionally, when using the StartStreaming command, if the channelInfos.receiveNotifications field (see the description of this input port) of the input tuple used to request these values was set to true, tuples will be emitted periodically to report on the monitored nodes' values. Thus, by controlling the value of the channelInfos.receiveNotifications field you can use this adapter to act either as a way to retrieve the value of multiple nodes in one call (channelInfos.receiveNotifications = false) or as a way to subscribe to the node(s)' ongoing changes (channelInfos.receiveNotifications = true).

The data field of each tuple emitted from this port is a list of all the nodes for which data was retrieved, their value, any optional attributes, and the like.

Field Name			Field Type	Description
header			tuple	The header of the ETP message represented by this tuple.
^	protocol		int
	messageType		int
	correlationId		long
	messageId		long
	messageFlags		int
data			list<tuple>	The data of the nodes retrieved.
^	channelID		long	ID of the node represented by this entry.
	indexes		list<long>
	value		string	The value of this node, represented as a string.
	valueAttributes		list<tuple>	The list of key-value pairs representing any additional attributes for this resource.
	^	attributeID	string
	^	attributeValue	string

Data Conversion

Because different nodes may contain values of different types, the various ETP operators represent each value as a string, but also provide metadata such as data type to inform on their actual representation in the ETP server. Using this information, the StreamBase application can parse the string values into more representative types.

Typechecking and Error Handling

Typechecking fails if the input schema does not contain all the required fields, or if a valid ETP Server Definition has not been specified.

Suspend and Resume Behavior

Upon suspension, the ETP adapters finish processing the current tuple, outputs the result tuple(s), then pauses waiting for input.

Upon resumption, the ETP adapters continue processing with the next input tuple.

In either case, any database connection is not modified.

ETP Adapter Sample

The StreamBase installation includes a sample demonstrating the use of this operator. To load the sample in StreamBase, select File>Import Samples and Community Content and look under the IoT section for an entry called ETP Adapters.

Back to Top ^