High Availability Finite State Machine Sample

HA FSM Sample Overview

The HA FSM sample shows the use of a Finite State Machine (FSM) to deal with state transitions in an HA design pattern. The behavior of the sample is similar to the behavior of the HA Sample, but the implementation of the HA FSM sample is more powerful and complex. Also, unlike the HA sample, leadership status is not based on a setting in the startup script. The server that starts first is designated as the leader, and any server that starts next is the nonleader.

HA FSM is designed to maximize flexibility — it can be extended or modified to handle Hot/Hot, Hot/Warm, and Hot/Cold HA configurations. By default, the sample is configured for a Hot/Warm setup.

Applications

HA FSM runs the same simple application, app.ssql, on each of two StreamBase Server instances. At any given time, one server is designated as primary, and the other is designated as secondary. The servers coordinate their behavior to provide high-availability access to the sample application's output.

How the Sample Works

As with the HA Sample there are two StreamBase Server instances in a Hot/Warm high-availability cluster. Both servers receive all input and perform all calculations, and application logic is identical on both servers, except that that one of the servers (the server that is not the leader) does not send output downstream. On failure of the leader, the nonleader takes over and begins sending output.

NOTE: The server that starts first is designated as the leader even though one is designated primary and the other is designated secondary. When you run the sample you can run the script that starts the server designated as secondary first to see this behavior.

There are two important applications for handling high availability: one specifies the HA logic and the other specifies the application logic. These files are specified in the configuration file for HA FSM (ha-fsm.sbconf) as follows:

<application file="${APPSDIR}/HaController.sbapp" container="ha_custom">
...
<application file="${HA_CONTROL_APP}" container="ha_control_${APP_CONTAINER}">         

HA Logic

The first application (HaController.sbapp) is the monitoring and detection module: it monitors state changes and detects the failure of the primary server. It runs in the ha_custom container. The HA logic uses a Java operator (defined in PingOperator.java) to determine the leadership status of a server.

To modify the behavior of this application you can either change the application or modify the values in the ha_ftc.fsm.config file. (This configuration file is read by a CSV File Reader Input adapter as part of FaultToleranceControllerFsm.sbapp.)

Logic to Control the Application

The second application (specified as the value of HA_CONTROL_APP) is the application handler: it specifies the change in application behavior to occur when there is a change in leadership status. If the server is not the leader, or if it stops being the leader, this application causes app.ssql to disable dequeuing.

The common.cmd (common on UNIX) script sets the value of the HA_CONTROL_APP environment variable as hacontrolcontainer.sbapp and the value of APP_CONTAINER as app.

To modify the behavior to meet your particular requirements, modify or replace the applications (and possibly rename their containers).

Containers

When you run the scripts for HA FSM the containers on each server are system, app, ha_control_app, and ha_custom. (Note that these are the same containers that run when you run the custom script of the HA Sample.)

This Sample's Files

The top-level application and configuration files delivered with the HA FSM sample are described in the following table.

File Description
app.sql
  • A simple application that consumes an integer and displays the hostname on which it is running as part of the output.

  • Designed to show that dequeuing behavior is associated with the leadership status of a server.

  • To be replaced with a more complex application of your own design.

HaController.sbapp
  • Connects to system control stream for the container at startup.

  • Monitors the status of the servers.

  • Changes leadership status if necessary.

  • To change the HA logic modify this application or edit ha-ftc.fsm.config, which is in the Configuration folder of the sample project.

hacontrolcontainer.sbapp
  • Connects to system control stream for the container at startup.

  • Processes HA and container tuples from the control stream.

  • If this server is the initial leader (or is made the leader) enables dequeuing of output from app.ssql.

  • Change this application if you want to take a different action when the leadership changes on a server.

ha-fsm.sbconf
  • Specifies the files and containers for the HA and application logic.

  • Describes the environment variables that must be set in the external environment.

  • Specifies module parameters and search paths for modules and operator resources.

sbd.sbconf
  • Placeholder file required by StreamBase Studio.

Trace flags in the configuration files allow the use of tracing to explore the workings of the sample in detail.

Additional files in the project support the custom Java operator and the lower level operations of the HA logic.

The scripts delivered with the HA FSM sample are described in the following table.

Windows script UNIX script Purpose
buildhajar.cmd --
  • Used in Windows to build the jar that contains PingOperator.

-- hastatus
  • Used in UNIX to check the status of the servers.

startprimary.cmd startprimary
  • Starts the primary StreamBase Server instance on the port specified by PRIMARY_PORT in common.cmd.

  • If this is the first server to start, sets the leadership status to LEADER. Otherwise sets the leadership status to NONLEADER.

  • Calls the common script (common.cmd or common)

startsecondary.cmd startsecondary
  • Starts the secondary StreamBase Server instance on the port specified by SECONDARY_PORT in common.cmd.

  • If this is the first server to start, sets the leadership status to LEADER. Otherwise sets the leadership status to NONLEADER.

  • Calls the common script (common.cmd or common).

startfeedsim.cmd startfeedsim
  • Runs the sbfeedsim command with a comma-separated double URI syntax that addresses both primary and secondary servers at the same time.

  • Starts a feed simulation based on the schema of the app.sql application.

  • Begins feeding generated values to both servers.

  • Calls the common script (either basic-common.cmd or custom-common.cmd).

startdequeue.cmd startdequeue
  • Runs sbc dequeue with the comma-separated double URI syntax, which begins dequeuing from both servers at the same time. However, only the server designated as leader is processing tuples to be dequeued.

  • Calls the common script (common.cmd or common).

common.cmd common
  • An environment-setting script called by all scripts.

  • Defines settings for high-availability, such as the names and hosts of the servers in the pair.

Exploring the HA FSM Sample in StreamBase Studio

StreamBase Studio can run only one server instance at a time, and the HA FSM sample does not make much sense unless both server instances are running. For this reason, you must run the sample in terminal or StreamBase Command Prompt windows.

However, you are encouraged to load each EventFlow application into StreamBase Studio to see how each is assembled.

Note

This sample makes use of environment variables and configuration file global parameters set with the <operator-parameter> element of the server configuration file. See Using Global Parameters in the Authoring Guide for more information.

Global parameters can be used by operators or adapters so that values are not hard-coded in the application. Global parameters that are referenced within an operator must be defined in the configuration file. In the configuration file, you can set the value of a global parameter by means of an environment variable. Putting these two features together allows you to define an global parameter value at runtime.

If you run this sample with the provided scripts, the environment variables are set for you. When experimenting with the sample in StreamBase Studio, or when running the sample without the provided scripts, see the comments in ha-fsm.sbconf for information on the environment variables that must be set.

Running the Samples

To get the sample up and running, you use scripts that do the following:

  • Start the servers.

  • Start a feed simulation to all streams.

  • Start a dequeue from all streams.

The scripts to run HA FSM are in the scripts folder.

Dequeuing from the sample application (app.ssql) is initially disabled on both servers. Shortly after startup, hacontrolcontainer.sbapp on the leader enables dequeuing for app.ssql.

You can start the sample by running either the startprimary or the startsecondary script first, and then running the other one. The server that starts first is designated as the leader, and the server that starts next is the nonleader. This is an important difference between the HA sample and the HA FSM sample.

Note

When you run the HA FSM sample, you may see warnings in the console for primary or secondary, such as:

2014-091-15 11:19:06.620-0400 [Thread- ThreadPool - 2] 
WARN com.streambase.sb.s d.net.HTTPServer - Connection /127.0.0.1:4666: I/O exception: 
An established connection was aborted by the software in your host machine 

Such warnings are not unusual and can be safely ignored.

To see the high availability features, stop the leader and observe the status of the other server.

Note

The detailed steps for the HA FSM sample are very similar to those for the HA Sample.

Importing This Sample into StreamBase Studio

In StreamBase Studio, import this sample with the following steps:

  • From the top menu, select FileLoad StreamBase Sample.

  • Select the hafsm sample from the High Availability category.

  • Click OK.

StreamBase Studio creates a single project containing the sample files. The default project name is sample_hafsm.

Sample Location

When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.

Important

Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.

Using the workspace copy of the sample avoids the permission problems that can occur when trying to work with the initially installed location of the sample. The default workspace location for this sample is:

studio-workspace/sample_hafsm

See Default Installation Directories for the location of studio-workspace on your system.

In the default TIBCO StreamBase installation, this sample's files are initially installed in:

streambase-install-dir/sample/hafsm

See Default Installation Directories for the location of streambase-install-dir on your system. This location may require administrator privileges for write access, depending on your platform.