Contents
The Spotfire Streaming Syslog Input Adapter allows a StreamBase application to act as a Syslog collector for receiving syslog messages.
The adapter has the following capabilities:
-
Listens for syslog messages sent to the given port number.
-
When such a message is received, outputs a tuple describing the values contained in the message.
-
Optionally receives commands from the StreamBase application to connect (start listening) and disconnect (stop listening).
-
Optionally issues status tuples on a dedicated to inform the application of interesting events such as connection and disconnection.
The adapter exposes a set of input and output ports, the fields for which are defined later in this document. In addition, the adapter exposes a set of adapter properties as described in Adapter Properties.
Settings are used to control most of the behavior of the adapter. The settings are grouped under several tabs in the Properties view in StreamBase Studio.
In the table in this section, the Property column shows each property name as found in the one or more adapter properties tabs of the Properties view for this adapter.
Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.
Adapter: A read-only field that shows the formal name of the adapter.
Class name: Shows the fully qualified class name that implements the functionality of this adapter. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.
Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.
Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.
Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.
| Property | Data Type | Default | Description |
|---|---|---|---|
| UDP port | int | 514 |
The port number to use, on which the adapter will listen for incoming
messages from Syslog agents.
NoteThis setting is required. |
| Parse Incoming Messages | boolean (check box) | true (selected) |
When selected, the adapter attempts to parse and interpret incoming
syslog messages in order to extract meaningful information (such as the
hostname of the machine originating the message, structured data
elements, etc). Otherwise the entirety of the message will be placed in
the rawMessage field (along with a timestamp
in the receiptTime field) and all other
fields will be left null. See Message Parsing Rules for details on the
parsing rules applied by the adapter.
|
| Connect On Startup | boolean (check box) | true (selected) | When selected, the adapter attempts to open the specified UDP port and start listening for incoming messages immediately after starting. If cleared, the adapter remains idle after starting and waits for a Connect command as described in Command Input Port. |
| Add Command Input Port | boolean (check box) | false (cleared) | When selected, the adapter adds an input port for the app to issue commands to the adapter, for example to make it start or stop listening for incoming messages. The schema for this port is described in Command Input Port. |
| Add Status Output Port | boolean (check box) | false (cleared) | When selected, the adapter adds an output port to issue events that may be of interest to the application, for example to inform the app that the adapter has started or stopped listening for incoming traps. The schema for this port is described in Status Output Port Schema. |
| Log Level | drop-down list | INFO | Controls the level of verbosity the adapter uses to send notifications to the console. This setting can be higher than the containing application's log level. If set lower, the system log level will be used. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE. |
Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.
Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.
Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.
Caution
Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.
The adapter has at most one input port, depending on its configuration (see Add Command Input Port in Adapter Properties Tab).
The (optional) Command input port is used to instruct the adapter to perform certain actions, such as connect or disconnect from the UDP port on which it listens for traps.
The schema for the Command Input port is expected to contain the following fields:
| Field Name | Field Type | Field Description |
|---|---|---|
command
|
string | Contains the name of the command to execute. This field is required to be non-null for all supported commands. See below for a list of the available commands. |
params
|
list(string) |
All-purpose list of string parameters that may be of use to specify
additional information to apply the command. Currently, this field is not
used by any of the supported commands and is only here to allow for
future expansion.
This field may be left null or empty if no parameters are required or needed. |
Available commands include:
-
Connect: Start listening for syslog messages on the configured UDP port. Required fields:
-
command: Should contain the valueConnect. Case is unimportant.
-
-
Disconnect: Stop listening for syslog messages. Required fields:
-
command: Should contain the valueDisconnect. Case is unimportant.
-
The adapter has at least one output port, and optionally a second one:
-
The first port issues a tuple for each syslog message received, and describes its content.
-
The second (optional) port routes status messages, such as connection and disconnection events.
The schema for each output port is described in the following sections:
The first output port routes incoming syslog messages to the StreamBase application.
The schema for this port is pre-set by the adapter, and is described in the following table.
Note
Only the rawMessage and receiptTime fields are guaranteed to be non-null in all cases.
The validity of the other fields in the schema will be dependent on whether the
adapter is configured to attempt further parsing of the message, and the extent
to which this parsing was successful. The parsing rules (and the fields they
affect) are described in Message Parsing Rules.
| Field Name | Field Type | Description | |||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| rawMessage | string | The entire syslog message, verbatim. This field will never be null. | |||||||||||||||||||||||||||
| receiptTime | timestamp | The time at which the message was received by the adapter. This field will never be null. | |||||||||||||||||||||||||||
| priority | int | The priority value of this message. This number is used to derive the originating facility as well as the severity of the message (specifically: facility * 8 + severity). | |||||||||||||||||||||||||||
| facility | int |
The facility that generated this message. Facilities are defined (in
RFC 5424, "The Syslog Protocol"
(March 2009)) as follows:
|
|||||||||||||||||||||||||||
| facility | int |
The severity of this message. Severities are defined as follows:
|
|||||||||||||||||||||||||||
| timestamp | timestamp | The time at which this event was generated. If the message conforms to RFC 5424, the string from which the timestamp is derived includes correct timezone information; if the message only conforms to RFC 3164, the string describing the timestamp did not include timezone information (since the spec makes no allowance for it) and the timestamp is interpreted as local time. | |||||||||||||||||||||||||||
| hostname | string | The host name or IP address of the originating system. | |||||||||||||||||||||||||||
| appname | string | The name of the device or program that generated the message. | |||||||||||||||||||||||||||
| procID | string | The ID of the process that originated the message. | |||||||||||||||||||||||||||
| msgID | string | The type of message. | |||||||||||||||||||||||||||
| msg | string | The actual message portion of this syslog entry. | |||||||||||||||||||||||||||
| structuredData | list<tuple> | Messages conforming to RFC 5424 may contain "structured data". This field contains the list of entries for such data. | |||||||||||||||||||||||||||
| ^ | id | string | Case-sensitive identifier for this entry. | ||||||||||||||||||||||||||
| ^ | params | list<tuple> | Contains the list of name/value pairs in this entry. | ||||||||||||||||||||||||||
| ^ | ^ | name | string | The name of this parameter. | |||||||||||||||||||||||||
| ^ | ^ | value | string | The value of this parameter. | |||||||||||||||||||||||||
The second output port routes Status messages to the
StreamBase application, provided the adapter has been configured to expose this
optional port (see Add Status Output Port in
Adapter Properties Tab).
The schema for this port is pre-set by the adapter, and is described in the following table:
| Field Name | Field Type | Field Description |
|---|---|---|
type
|
string |
Describes the type of status being emitted. Currently the only value
supported is Event.
|
action
|
string |
The action that triggered this status tuple. One of:
|
info
|
list(string) | Contains extra information pertaining to the event. May be null or empty depending on the type of event being described. |
Syslog messages do not have to conform to a hard and fast format. The only real requirement is that only one message be present in a given UDP packet. However over the years some conventions have emerged and some specifications have even been published, with varying degrees of adoption.
When the adapter receives a syslog message it will at a minimum set the rawMessage field of the outgoing tuple to contain the entirety of
this message, and the receiptTime field will contain a
time stamp indicating the exact time at which the message was received by the adapter
(this is not to be confused with the timestamps often contained within the message
itself, which represent the time at which the message was sent). These two fields are
the only ones that are guaranteed to be non-null and meaningful in all cases,
regardless of whether further interpretation has been attempted and was successful.
When the adapter is configured to attempt further parsing of the messages (see Adapter Properties Tab), incoming syslog messages are examined in order to extract meaningful information (such as the hostname of the machine originating the message, RFC 5424 "structured data elements", etc). If identified in the parsing process, these discrete elements are placed in dedicated fields in the output schema to aid the application in interpreting and characterizing the message.
Several parsing attempts are made, according to the following rules:
-
Parse using the specification outlined in RFC 5424, "The Syslog Protocol" (March 2009).
-
If the message's format does not conform to the above, parse it using the specification outlined in RFC 3164, "The BSD Syslog Protocol" (August 2001).
-
If the message still cannot be parsed successfully, attempt to at least extract the message's priority by checking if it begins with the pattern
<NNN>where NNN is an integer number. If successful, the priority is extracted and the rest of the message is treated as a single string. -
Failing all these attempts (or if Parse Incoming Messages is unchecked) the entirety of the message is placed in the
rawMessagefield and all other fields are left null.
The complete list of fields in the output schema describing a syslog message is given in Syslog Message Output Port Schema. The table below describes the circumstances in which a given field's value may be set. Note of course that even when a message conforms to a given format some fields may still be null if they had no value in the message (for example an RFC 5242 message may leave its 'appname' component unset while still providing meaningful values for other fields).
| Field name | Parsing rules from the above list that may set its value |
|---|---|
rawMessage
|
1, 2, 3, 4 |
receiptTime
|
1, 2, 3, 4 |
priority
|
1, 2, 3 |
facility
|
1, 2, 3 |
severity
|
1, 2, 3 |
timestamp
|
1, 2 |
hostname
|
1, 2 |
appname
|
1 |
procID
|
1 |
msgID
|
1 |
msg
|
1, 2, 3 |
structuredData
|
1 |
Devices and programs generating syslog messages do so by sending them to the system's Syslog daemon (syslogd). This daemon is the program responsible for properly routing the messages to local logs file and/or collectors such as the adapter. The syslogd daemon needs to be explicitly told to forward messages to your StreamBase application in order for the adapter to receive them. This is done by modifying the /etc/syslog.conf configuration file. To forward all syslog messages to your application, ensure the following line is included (replacing the host name by that of the machine running StreamBase):
*.* @logserv.example.com
(You will need root access to make this change.)
Afterwards, restart syslogd with the command:
service syslogd restart
From then on the client machine will forward all syslog entries to your StreamBase application. Refer to syslogd(8) and syslog.conf(5) documentation for instructions on configuring finer-grained forwarding rules (e.g. only forwarding messages of a certain priority).
The adapter uses typecheck messages to help you configure the adapter in your StreamBase application. In particular, the adapter generates typecheck messages for the following reasons:
-
A required property is missing.
-
One or more required fields in the Command input port is missing or is of the wrong type or size.
When suspended, the adapter remains connected to the configured UDP port if it was already connected, but all incoming traps are dropped.
When resumed, the adapter starts routing incoming traps to the StreamBase application again.