- Index

< Previous |  | Next >

# Using StreamBase Logging

## Logging Overview

For all supported operating systems, StreamBase uses the industry-standard SLF4J logging system with the Logback framework. See Windows Event Logging for additional considerations for Windows.

All subsystems in EventFlow and LiveView fragments use Logback logging. You can configure the level of logging information emitted, and can direct logs to different appenders using standard Logback configuration files in either XML or Groovy formats.

Log files are written by default into a subdirectory named `logs` in a fragment's node directory:

- For fragments run in StreamBase Studio, the node directory is removed on Studio exit for all successful fragment launches. If a launch fails, its node directory is preserved so that you can inspect log files.

- For fragment launches made from the command line, you specify the location of each fragment's node directory as part of the epadmin node install command. In this case, the node directory and its `logs` subdirectory is preserved in the location you specify.

To configure a custom Logback configuration files see Custom Configuration and StreamBase Logback Extensions below. The default `logs` subdirectory location can be overridden as described in StreamBase Logback Extensions.

## Logging Configuration Processing Order

The StreamBase Runtime processes log configuration in the following order:

1. Default logging.

2. Logging configuration specified in fragment archives generated from fragment projects.

3. Logging configuration specified in StreamBase Application archives generated from an Application project.

Logging configurations do not merge. That is, a later configuration in the processing order sequence overrides an earlier one. For this reason, any custom logging configuration file should specify settings for both RootFileAppender and RootConsoleAppender to avoid error messages.

Logging settings are ultimately specified per engine. If your fragment has more than one engine, each engine can inherit the settings made for the fragment and application as a whole.

The logging level and logging configuration can be changed at runtime per engine using the logging target of the epadmin command, as described in Managing Logging at Runtime.

## The Log Output Adapter

In addition to the logging messages emitted by EventFlow and LiveView as a whole, you can add logging instructions to any EventFlow module with the Log Output adapter. The adapter has a sample described in Log Output Adapter Sample that illustrates formatting options you can perform with the Log Output adapter.

Do not confuse the Java logging facility discussed here with the StreamBase error stream feature. Logging is for writing your own logging messages and exception stack traces to one or more log files. By contrast, the error stream feature is like a module-level throw that ascends up the call stack to the containing module, and from there to the top-level EventFlow module instance running the fragment. StreamBase sends an error tuple to the error stream when it encounters an exception during a tuple processing method. See Using Error Ports and Error Streams for further information.

## Logging in Custom Code

If you write custom operators or adapters, you can use an `import org.slf4j.Logger` statement in your code to define a logger for your operator or adapter. The base class, `Operator`, provides the convenience method `getLogger()`, which returns an `org.slf4j.Logger` object that should thereafter be used for logging messages and exceptions in your code. For the full list of methods available on `Logger` objects, refer to the Logger class in the SLF4J Javadoc.

The StreamBase installation includes a logging sample that illustrates the difference between using a Log Output adapter and using `getLogger()` methods in your custom code. This sample compares both methods with a third module that does not log.

## Logback Log Levels

StreamBase uses the standard log level meanings provided by the Logback framework, as defined in the Logger API.

Method | Notes
--- | ---
ERROR | A node level error caused by a resource problem (memory, disk, file descriptors, and so on), a corrupted product installation with missing required resources, or host machine failures. The node is terminated and must be reinstalled to restore service.
WARN | An operational error that impacts a specific operation, application performance, or availability of certain services. An operational error is caused by invalid parameters, temporary resource unavailability, or authentication and/or authorization failures. No other node services are affected. The node may automatically recover from this error, but in some cases operator intervention is required for full recovery.
INFO | A normal status message for node and engine life cycle, resource start-up information, and environment details such as library version numbers, class path contents, or location of configuration files. No action is required.
DEBUG | Mid-level diagnostic debug messages. DEBUG messages should only be enabled on production systems at the request of TIBCO Technical Support. These messages are only of interest to the product development team.
TRACE | Low-level diagnostic trace messages. TRACE messages should only be enabled on production systems at the request of TIBCO Technical Support. These messages are only of interest to the product development team.

## Legacy Log Levels

The following table maps the log level integer settings used in StreamBase releases before 10.2.0 to Logback message levels.

Level | Method | Notes
--- | --- | ---
–2 | ERROR |
–1 | WARN |
0 | INFO | The default setting. (Legacy StreamBase releases called this level `notice`.)
1 or 2 | DEBUG | Levels 1 and 2 are equivalent ways to specify DEBUG level logging, and produce the same number of log messages.
3 | TRACE | (Legacy StreamBase releases called this level `debug2`.)

## Separate Adapter Log Levels

Most StreamBase adapters allow you to set a log level independently for the adapter than for the hosting EventFlow module:

- The Wombat MAMA Input adapter has a set of log levels that derive from the Wombat MAMA API.

- Most other adapters allow you to set the log level for log messages emitted from that adapter. To determine whether an adapter of interest has independent logging, look for a Log Level property in the Adapter Properties section of the documentation page for that adapter in the Adapters Guide.

For example, an adapter might set the log level to WARN or ERROR for messages emitted from the adapter while the hosting fragment is still set to its default INFO level. This allows you to troubleshoot adapter connectivity issues without having to wade through hundreds of verbose server messages from the fragment.

## Logback Default Configuration

StreamBase's default settings for Logback logging are as follows:

- Two appenders are configured, the console appender and a file appender that writes to the `logs` subdirectory of the node directory.

- Log messages from third-party libraries used in StreamBase are set to minimize their chattiness.

- The default log level is INFO.

- The default log message pattern is internally defined as:

      %d{yyyy-MM-dd HH:mm:ss.}%usecs%d{Z} [%process:%thread] %-5level %logger: %msg%n

- One log file is written per engine in a running fragment, with the log file name ``enginename`.log`.

- Log file rollover is set to a maximum size of 10 MB for ten files.

For fragments launched from Studio, the `logs` subdirectory includes three log file types:

- `.log` extension — log messages from the StreamBase Runtime

- `.out` extension — log messages sent to standard output

- `.err` extension — log messages sent to standard error

The components of the default log message pattern have the meanings shown in the following table.

Message Pattern Component | Meaning
--- | ---
%d{yyyy-MM-dd HH:mm:ss} | Formatted timestamp.
%usecs%d{Z} | Adds the microseconds and time zone values to the timestamp. (%usecs is a StreamBase extension.)
[%process:%thread] | The process ID and the name of the thread that generated the logging event. (%process is a StreamBase extension.)
%-5level | The log level in five characters, left-justified, with one or more padding spaces added on the right.
%logger | The qualified name of the class or component from which the logging event was emitted
%msg%n | Everything after the hyphen represents the text of the log message emitted by an instance of the Logger class in code. %n represents the line ending character or characters appropriate for the current operating system.

Consult the Logback documentation for PatternLayout for the available log message pattern components.

You can override any of the default settings with a custom Logback configuration.

## Custom Configuration for Logback Logging

To configure Logback logging, specify a Logback configuration file. StreamBase accepts files in either the traditional XML format, often named `logback.xml`, or in the alternative Groovy format with `.groovy` extension. In either case, place your configuration files in the `src/main/resources` folder of your Studio project.

You can also use the `src/test/resources` folder for a logging configuration you only need when running Maven tests. To have one configuration used both for standard fragment launches and tests, use `src/test/resources`.

The StreamBase `logging` sample includes an example `logback.xml` configuration file that illustrates a setup for both a console appender and a rolling file appender.

## Logback Configuration using Groovy

The Groovy language format for Logback configuration files is described on https://logback.qos.ch/manual/groovy.html. One of the advantages of defining your configuration using Groovy is that you can introduce conditional statements. For example, you can specify a certain console appender format when running on all development hosts, but not on production hosts.

The Logback site provides an XML to Groovy translator at https://logback.qos.ch/translator/asGroovy.html.

## StreamBase Logback Extensions

In general, StreamBase Logback configuration files use standard Logback patterns as described in the Logback documentation for PatternLayout. StreamBase provides the following extensions to the standard Logback patterns:

%usec

  A pattern that specifies the microsecond portion of the current second, used with a timestamp pattern:

      %d{yyyy-MM-dd HH:mm:ss.}%usecs%d{Z}

%process

  A pattern that specifies the process ID of the currently running engine.

      [%process:%thread]

com.tibco.ep.dtm.logging.logDirectory

  A Logback substitution variable with a default value of ``node-directory`/logs`.

com.tibco.ep.dtm.logging.logFileNamePrefix

  A Logback substitution variable with a default value of the current engine name.

You can define different values for these variables in your configuration file:

    <variable name="com.tibco.ep.dtm.logging.logDirectory" value="/Users/sbuser/logs"/>
    <variable name="com.tibco.ep.dtm.logging.logFileName" value="testrun"/>
    ...
    <appender name="FILE" class="ch.qos.logback.core.FileAppender">
      <file>${com.tibco.ep.dtm.logging.logDirectory}/${com.tibco.ep.dtm.logging.logFileNamePrefix}.log>/file>
      <encoder>
    . ...

Search for "Variable substitution" on this Logback documentation page to learn more about Logback variables.

## Command Audit Log

StreamBase creates entries in an audit log for every epadmin command issued per node, stored by default in the same `logs` subdirectory of the node directory.

The audit log:

- Is named `audit.log`.

- Has a default log level of INFO.

- Has the following Logback message format:

      %d{yyyy-MM-dd HH:mm:ss.SSSZ}: %msg%n

Command audit messages are saved only in the audit log, and are not duplicated in the other node logs.