High Availability Sample

Contents

HA Sample Overview
This Sample's Files
Exploring the HA Sample in StreamBase Studio
Running This Sample with StreamBase Authentication
Running This Sample in Terminal Windows
Running the HA Heartbeat Tester Sample
Importing This Sample into StreamBase Studio
Sample Location
Related Topics

HA Sample Overview

The HA sample shows how to enable automatic high availability for two StreamBase Server instances running together in a Hot/Warm high-availability cluster. The sample runs the same simple application in two servers, one designated primary and the other designated secondary. There are two parts to the sample: a basic sample and custom sample. There is also an application that you can run to examine how the HA Heartbeat adapter works.

  • The basic sample is set of applications and scripts where automatic leadership control is enabled.

  • The custom sample is similar, except that it does not enable automatic leadership control. Instead it uses a container to monitor HA leadership status of the servers. The purpose of the custom sample is to show details to help you customize leadership control to meet your own requirements.

The behavior of the two samples when you run them is the same.

Applications

Each sample runs the same simple application, app.ssql, on each of two StreamBase Server instances. At any given time, one server is designated as primary, and the other is designated as secondary. On startup the primary server is designated as leader, and the secondary is designated as nonleader. The servers coordinate their behavior to provide high-availability access to the sample application's output.

How the Basic Sample Works

When automatic HA is enabled, at runtime StreamBase creates a container that runs an application that determines if the other server in a HA pair has stopped running; this container is named _SB_ha and the application is haleadershipcontrol.sbapp. For example, if automatic HA is enabled for app.ssql, StreamBase creates three containers: app, system, and _SB_ha, on each server.

The containers are shown in the following illustration:

The automatic HA application, haleadershipcontrol.sbapp, uses a Heartbeat adapter to determine if the other server in a pair has stopped running. The other server in the pair is specified as a property of the HA Heartbeat adapter; the setting is determined by means of a setting in the ha.sbconf file.

Note

See Automatic HA for a description of settings to enable automatic HA.

Besides app.ssql, the basic sample includes hacontrolcontainer.sbapp, which is not provided automatically by the StreamBase automatic HA feature (it is not in the illustration above). This application controls the behavior of the app container on its server, according to the leadership status of the server, as determined by haleadershipcontrol.sbapp. If the server is not the leader, or if it stops being the leader, hacontrolcontainer.sbapp causes app.ssql to disable dequeuing.

Thus, when you run the scripts for the basic sample there are four containers on each server (system, app, ha_control_app, and _SB_ha).

The containers are shown in the following illustration:

You can customize the basic sample by modifying, replacing, or extending hacontrolcontainer.sbapp, while still using the automatic HA feature.

Note

For purposes of the HA sample, the apps folder in the Package Explorer includes haleadershipcontrol.sbapp as part of the StreamBase Studio project. You can open it in the EventFlow Editor to understand the StreamBase logic. However, do not modify, extend, or replace hacontrolcontainer.sbapp unless you switch off automatic HA.

How the Custom Sample Works

The configuration file for the custom sample is ha-custom.sbconf, which includes the following line: 

<application file="${HA_MANUAL_APP}" container="ha">

where HA_MANUAL_APP is the name of the application that monitors leadership changes. The common.cmd (common on UNIX) script sets the value of the HA_MANUAL_APP environment variable. Like the basic sample, the custom sample uses haleadershipcontrol.sbapp as the application that monitors leadership status.

The configuration file specifies ha_custom as the name of the container running haleadershipcontrol.sbapp. Notice that this name differs from the name assigned automatically (_SB_ha) when automatic HA is enabled.

Thus, when you run the scripts for the custom sample the four containers on each server are system, app, ha_control_app, and ha_custom.

The containers are shown in the following illustration:

When you run the scripts for the custom sample haleadershipcontrol.sbapp is used but it was not created automatically. Because automatic HA is not enabled, you can modify, replace, or extend haleadershipcontrol.sbapp as well as hacontrolcontainer.sbapp. That is, in the custom sample, you can customize the application logic for both the detection and the handling of a change in leadership status.

Running the Samples

The basic and custom HA samples each have scripts that are used to run the sample; they are in the basic-scripts and custom-scripts folders, respectively. The scripts differ only in terms of the configuration files that they specify.

Dequeuing from the sample application (app.ssql is initially disabled on both servers. Shortly after startup, hacontrolcontainer.sbapp on the leader (the primary server) enables dequeuing for app.ssql.

This Sample's Files

The application and configuration files delivered with the HA sample are described in the following table.

File Description
app.sql

  • A simple application that consumes an integer and displays the hostname on which it is running as part of the output.

  • Designed to show that dequeuing behavior is associated with the leadership status of a server.

  • To be replaced with a more complex application of your own design.

haleadershipcontrol.sbapp

  • Connects to system control stream for the container at startup

  • Monitors the status of the secondary server by means of an HA Heartbeat adapter.

  • Changes leadership status on its server if necessary.

hacontrolcontainer.sbapp

  • Connects to system control stream for the container at startup.

  • Processes HA and container tuples from the control stream.

  • If this server is the initial leader (or is made the leader) enables dequeuing of output from app.ssql.

  • Change this application if you want to take a different action when the leadership changes on a server.

RunSBAdminCommand.sbapp

  • Defines an external process operator, which handles StreamBase commands that are issued when running the sample, and also does some error checking.

ha.sbconf

  • Used only by the basic sample.

  • Server configuration file used by both primary and secondary servers in the basic part of the sample.

  • The <high-availability> section specifies that automatic leadership control is enabled, along with parameters required to support high-availability processing.

ha-custom.sbconf

  • Used only by the custom sample.

  • Disables automatic leadership control and uses customized HA instead.

  • Specifies the HA application and the name of the container to use for leadership control.

sbd.sbconf

  • Server configuration file

  • Provides StreamBase Studio with the names of environment variables used in the HA application.

hbtester.sbapp

  • Standalone application, independent of the basic and common samples.

  • Use this application to test and learn about the HA Heartbeat adapter.

Trace flags in the configuration files allow the use of tracing to explore the workings of the sample in detail.

There are two sets of scripts delivered with the HA sample; one set is in the basic-scripts folder and the other is in the custom-scripts folder. The scripts are described in the following table.

Windows script UNIX script Purpose
startprimary.cmd startprimary

  • Starts the primary StreamBase Server instance on the port specified by PRIMARY_PORT in common.cmd. Sets the leadership status to LEADER.

  • Calls the common script (either basic-common.cmd or custom- common.cmd)

startsecondary.cmd startsecondary

  • Starts the secondary StreamBase Server instance on the port specified by SECONDARY_PORT in common.cmd. Sets the leadership status to NON_LEADER.

  • Calls the common script (either basic-common.cmd or custom-common.cmd).

startprimaryasnonleadercmd startprimaryasnonleader

  • Starts the StreamBase Server instance on the port specified by PRIMARY_PORT in common.cmd.

  • Sets the leadership status to NON_LEADER.

  • Calls the common script (either basic-common.cmd or custom-common.cmd).

  • Use this script to restart the primary server after it has failed and the secondary server has become the leader.

startfeedsim.cmd startfeedsim

  • Runs the sbfeedsim command with a comma-separated double URI syntax that addresses both primary and secondary servers at the same time.

  • Starts a feed simulation based on the schema of the app.sql application.

  • Begins feeding generated values to both servers.

  • Calls the common script (either basic-common.cmd or custom-common.cmd).

startdequeue.cmd startdequeue

  • Runs sbc dequeue with the comma-separated double URI syntax, which begins dequeuing from both servers at the same time. However, only the server designated as leader is processing tuples to be dequeued.

  • Calls the common script (either basic-common.cmd or custom-common.cmd).

common.cmd common

  • An environment-setting script called by all scripts.

  • Defines settings for high-availability, such as the names and hosts of the servers in the pair.

basic-common.cmd basic-common

  • Calls the scripts for the basic sample, which are in the basic-scripts folder.

  • Specifies ha.sbconf as the configuration file.

custom-common.cmd custom-common

  • Calls the scripts for the custom sample, which are in the custom-scripts folder.

  • Specifies ha-custom.sbconf as the configuration file.

Exploring the HA Sample in StreamBase Studio

StreamBase Studio can run only one server instance at a time, and the HA sample does not make much sense unless both server instances are running. For this reason, you must run the sample in terminal or StreamBase Command Prompt windows as described in the next section.

However, you are encouraged to load each EventFlow application into StreamBase Studio to see how each is assembled.

Note

This sample makes use of environment variables and configuration file global parameters set with the <operator-parameter> element. See Using Global Parameters in the Authoring Guide for more information.

Global parameters can be used by operators or adapters so that values are not hard-coded in the application. Global parameters that are referenced within an operator must be defined within the configuration file. In the configuration file, you can set the value of a global parameter by means of an environment variable. Putting these two features together allows you to define an global parameter value at runtime.

If you run this sample with the provided scripts, the environment variables are set for you. When experimenting with the sample in StreamBase Studio, or when running the sample without the provided scripts, see the comments in ha.sbconf and ha-custom.sbconf for information on the environment variables that must be set.

Running This Sample with StreamBase Authentication

The HA sample is shipped with support for StreamBase authentication disabled. To run this sample with StreamBase authentication enabled you must perform the following prerequisite steps:

  1. Create an authentication database, usually stored in $STREAMBASE_HOME/etc, using the sbuseradmin command. For instructions, see Using StreamBase Server Simple Authentication.

  2. In the authentication database, create at least one StreamBase username with the SBAdmin role.

  3. Set the following environment variables:

    AUTH true
    AUTH_PASSWORD_FILE Path to the authentication password file created with sbuseradmin.
    AUTH_ADMINUSER Username assigned the SBAdmin role in the authentication file.
    AUTH_ADMINPASSWORD Password for the above username.

    These variables apply to the HA family of samples, and are not StreamBase standard environment variables.

Running This Sample in Terminal Windows

This section shows how to run the samples and suggests some steps to show how the servers, the containers, and the applications work together to ensure high availability. You can, of course, experiment with other steps of your own.

To get each sample up and running you use scripts that do the following:

  • Start the servers.

  • Start a feed simulation to all streams.

  • Start a dequeue from all streams.

The scripts are in one or the other of two directories according to which sample you want to run.

Because StreamBase Studio can run only one server instance at a time, you must run this sample's two servers in terminal or StreamBase Command Prompt windows.

Note

On Windows, be sure to use the StreamBase Command Prompt from the Start menu as described in the Test/Debug Guide, not the default command prompt.

The steps to run the sample are as follows:

  1. Set up five terminal windows.

    1. Open five terminal windows on UNIX, or five StreamBase Command Prompts on Windows.

    2. In each window, navigate to your workspace copy of the sample.

      For the basic sample, navigate to the basic-scripts folder; for the custom sample, navigate to the custom-scripts folder.

      Note

      The steps are the same for each sample. Navigate first to the basic-scripts folder, follow the steps for running the sample, then navigate to the custom-scripts folder and repeat the steps.

  2. Start the following processes:

    1. In window 1, start the first server with this command: startprimary.

      This script sets environment variables, starts a server listening on port 9900, and loads the high availability application (ha) in a container, ha, on that server. The high availability application designates this server as the leader and starts app.ssql.

      In window 5, run the following command to observe this outcome: 

      sbc -u :9900 list containers

      This command generates the following output: 

      container   _SB_ha
container   app
container   ha_control_app
container   system

    2. In window 2, run startsecondary.

      This script performs the same actions as startprimary, except it starts its server on port 9901 and sets its leadership status as nonleader.

    3. In window 3, run startfeedsim.

      A feed simulation starts sending input data to both servers. Window 3 shows input, such as the following: 

      in (a=871)
in (a=6419)
in (a=2348)
in (a=8339)
in (a=5707)
in (a=909)
in (a=8756)
in (a=3983)
in (a=8748)
...

    4. In window 4, run startdequeue.

      The command sent by the startdequeue script is a comma-separated list of URIs to address both servers at the same time. After resolving the variables for port number, the command sent by startdequeue resolves to the following: 

      sbc -u sb://localhost:9900/app,sb://localhost:9901/app dequeue

      Notice the hostname (myhostname) and the StreamBase URI of the responding server is shown for each line, and only one server is responding, namely the server designated as the leader. 

      out,2555,myhostname,sb://localhost:9900/
out,3091,myhostname,sb://localhost:9900/
out,8482,myhostname,sb://localhost:9900/
out,520,myhostname,sb://localhost:9900/
out,101,myhostname,sb://localhost:9900/
out,3538,myhostname,sb://localhost:9900/
out,867,myhostname,sb://localhost:9900/
out,8503,myhostname,sb://localhost:9900/
out,36,myhostname,sb://localhost:9900/
...

  3. Test the high-availability features of the sample:

    1. In window 5, run the following command to determine the status of both servers: 

      sbc -u :9900,:9901 status --verbose

      This command generates a status report line for each server. Notice that the server on port 9900, which is running in window 1, is designated as the leader.

    2. In window 5, run the following command to shut down the primary server, the one that is now the leader: 

      sbadmin -p 9900 shutdown

      Notice that the dequeued output stream in window 4 has switched to showing tuples from the server on port 9901: 

      out,6700,myhostname,sb://localhost:9901/
out,7156,myhostname,sb://localhost:9901/
out,1303,myhostname,sb://localhost:9901/
out,4798,myhostname,sb://localhost:9901/
out,2305,myhostname,sb://localhost:9901/
out,9540,myhostname,sb://localhost:9901/
...

    3. In window 5, run the following command to determine the status of both servers: 

      sbc -u :9900,:9901 status --verbose

      Notice that the server on port 9901, which is running in window 2, is designated as the leader.

    4. In window 1, restart the primary server by running the startprimaryasnonleader command. Notice that this server is the nonleader.

    5. Shut down the secondary server, the one that is now the leader: 

      sbadmin -p 9901 shutdown

      Notice the output stream in window 4 switches back to the server on port 9900.

    6. Check the status again by running the same command as before: 

      sbc -u :9900,:9901 status --verbose

      Now, only one server is running, namely the server on port 9900, and it is the leader.

  4. To shut down all servers, run the following commands

    • In window 4, press Ctrl+C to stop the dequeuer.

    • In window 3, press Ctrl+C to stop the feed simulation.

    • In window 5, run: 

      sbadmin -p 9900 shutdown
sbadmin -p 9901 shutdown

  5. Repeat the steps for the custom sample.

    For the custom sample navigate to the following directory to run the scripts: 

    sample_ha/custom-scripts

  6. Optionally, experiment with the haleadershipcontrol.sbapp to try out different strategies for customized leadership control.

Running the HA Heartbeat Tester Sample

This section describes how to run the hbtester.sbapp application to learn about the interaction of two instances of the HA Heartbeat adapter and test different scenarios for using it. This application runs standalone, independent of the other applications in this sample.

Because StreamBase Studio can run only one server instance at a time, you must run this sample's two instances in terminal or StreamBase Command Prompt windows.

The hbtester.sbapp application can be run as either client or server heartbeat adapter. Run them on separate ports to run them at the same time, using commands like the following:

  1. Set up six terminal windows.

    1. Open six terminal windows on UNIX, or six StreamBase Command Prompts on Windows.

    2. In each window, navigate to your workspace copy of the sample.

  2. In window 1, start the first HA Heartbeat instance: 

    sbd -p 9000 hbtester.sbapp

  3. In window 2, start the second instance: 

    sbd -p 9010 hbtester.sbapp

  4. In windows 3 and 4, start dequeuers for each server: 

    sbc -p 9000 dequeue
sbc -p 9010 dequeue

  5. In windows 5 and 6, start enqueuers for each server: 

    sbc -p 9000 enq CommandIn
sbc -p 9010 enq CommandIn

  6. In windows 5 and 6, experiment with various commands to activate and deactivate the two HA Heartbeat adapters, watching the resulting output messages in windows 3 and 4. For example, try the following sequence:

    • In window 5, enter: startServer

    • No output is seen yet because no heartbeat clients have yet started. Now in window 6, enter: startClient

    • Look for output messages in windows 3 and 4 from both servers.

    • In window 6, enter: stop

    • Look for a disconnected message in window 3 and a timeout message in window 4.

    • In window 6, enter: start

    • Look for the output messages to resume in windows 3 and 4.

    • In windows 5 and 6, stop both server and client heartbeat adapters by entering: stop

    • Reverse the positions of the server and client heartbeats. In window 6, enter: startServer

    • In window 5, enter: startClient

    • Look for the resumption of output messages in windows 3 and 4.

    Continue experimenting as you like, paying attention to the content and format of output messages in windows 3 and 4. This will help you design your HA-compliant application to respond to messages from the HA Heartbeat under different circumstances.

  7. When done:

    1. In windows 5 and 6, type Ctrl+C to exit both sbc enq sessions.

    2. In another terminal window or StreamBase Command Prompt, type: 

      sbadmin -p 9000 shutdown
sbadmin -p 9010 shutdown

      This terminates both servers and both dequeuers, and closes windows 1 through 4.

See HA Heartbeat Input Adapter for details on the operation of the HA Heartbeat adapter.

Back to top ^

Importing This Sample into StreamBase Studio

In StreamBase Studio, import this sample with the following steps:

  • From the top menu, select File → Load StreamBase Sample.

  • Select the ha sample from the High Availability list.

  • Click OK.

StreamBase Studio creates a single project containing the sample files. The default project name is sample_ha.

Sample Location

When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.

Important

Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.

Using the workspace copy of the sample avoids the permission problems that can occur when trying to work with the initially installed location of the sample. The default workspace location for this sample is: 

studio-workspace/sample_ha

See Default Installation Directories for the location of studio-workspace on your system.

In the default TIBCO StreamBase installation, this sample's files are initially installed in: 

streambase-install-dir/sample/ha

See Default Installation Directories for the location of streambase-install-dir on your system. This location may require administrator privileges for write access, depending on your platform.

Related Topics

Automatic HA
Clustering and High Availability
Designing Applications for High Availability
Query Table Replication Sample