Contents
The HA sample shows how to enable automatic high availability for two StreamBase Server instances running together in a Hot/Warm high-availability cluster. The sample runs the same simple application in two servers, one designated primary and the other designated secondary. There are two parts to the sample: a basic sample and custom sample. There is also an application that you can run to examine how the HA Heartbeat adapter works.
-
The basic sample is set of applications and scripts where automatic leadership control is enabled.
-
The custom sample is similar, except that it does not enable automatic leadership control. Instead it uses a container to monitor HA leadership status of the servers. The purpose of the custom sample is to show details to help you customize leadership control to meet your own requirements.
The behavior of the two samples when you run them is the same.
Each sample runs the same simple application, app.ssql
, on each of two StreamBase Server instances. At any given time, one server is designated as primary, and the other is designated
as secondary. On startup the primary server is designated as leader, and the secondary is designated as nonleader. The servers
coordinate their behavior to provide high-availability access to the sample application's output.
When automatic HA is enabled, at runtime StreamBase creates a container that runs an application that determines if the other
server in a HA pair has stopped running; this container is named _SB_ha
and the application is haleadershipcontrol.sbapp
. For example, if automatic HA is enabled for app.ssql
, StreamBase creates three containers: app
, system
, and _SB_ha
, on each server.
The containers are shown in the following illustration:
The automatic HA application, haleadershipcontrol.sbapp
, uses a Heartbeat adapter to determine if the other server in a pair has stopped running. The other server in the pair is
specified as a property of the HA Heartbeat adapter; the setting is determined by means of a setting in the ha.sbconf
file.
Note
See Automatic HA for a description of settings to enable automatic HA.
Besides app.ssql
, the basic sample includes hacontrolcontainer.sbapp
, which is not provided automatically by the StreamBase automatic HA feature (it is not in the illustration above). This application controls the behavior of the app
container on its server, according to the leadership status of the server, as determined by haleadershipcontrol.sbapp
. If the server is not the leader, or if it stops being the leader, hacontrolcontainer.sbapp
causes app.ssql
to disable dequeuing.
Thus, when you run the scripts for the basic sample there are four containers on each server (system
, app
, ha_control_app
, and _SB_ha
).
The containers are shown in the following illustration:
You can customize the basic sample by modifying, replacing, or extending hacontrolcontainer.sbapp
, while still using the automatic HA feature.
Note
For purposes of the HA sample, the apps
folder in the Package Explorer includes haleadershipcontrol.sbapp
as part of the StreamBase Studio project. You can open it in the EventFlow Editor to understand the StreamBase logic. However,
do not modify, extend, or replace hacontrolcontainer.sbapp
unless you switch off automatic HA.
The configuration file for the custom sample is ha-custom.sbconf
, which includes the following line:
<application file="${HA_MANUAL_APP}" container="ha">
where HA_MANUAL_APP is the name of the application that monitors leadership changes. The common.cmd
(common
on UNIX) script sets the value of the HA_MANUAL_APP environment variable. Like the basic sample, the custom sample uses haleadershipcontrol.sbapp
as the application that monitors leadership status.
The configuration file specifies ha_custom
as the name of the container running haleadershipcontrol.sbapp
. Notice that this name differs from the name assigned automatically (_SB_ha
) when automatic HA is enabled.
Thus, when you run the scripts for the custom sample the four containers on each server are system
, app
, ha_control_app
, and ha_custom
.
The containers are shown in the following illustration:
When you run the scripts for the custom sample haleadershipcontrol.sbapp
is used but it was not created automatically. Because automatic HA is not enabled, you can modify, replace, or extend haleadershipcontrol.sbapp
as well as hacontrolcontainer.sbapp
. That is, in the custom sample, you can customize the application logic for both the detection and the handling of a change
in leadership status.
The basic and custom HA samples each have scripts that are used to run the sample; they are in the basic-scripts
and custom-scripts
folders, respectively. The scripts differ only in terms of the configuration files that they specify.
Dequeuing from the sample application (app.ssql
is initially disabled on both servers. Shortly after startup, hacontrolcontainer.sbapp
on the leader (the primary server) enables dequeuing for app.ssql
.
The application and configuration files delivered with the HA sample are described in the following table.
File | Description |
---|---|
app.sql |
|
haleadershipcontrol.sbapp |
|
hacontrolcontainer.sbapp |
|
RunSBAdminCommand.sbapp |
|
ha.sbconf |
|
ha-custom.sbconf |
|
sbd.sbconf |
|
hbtester.sbapp |
|
Trace flags in the configuration files allow the use of tracing to explore the workings of the sample in detail.
There are two sets of scripts delivered with the HA sample; one set is in the basic-scripts
folder and the other is in the custom-scripts
folder. The scripts are described in the following table.
Windows script | UNIX script | Purpose |
---|---|---|
startprimary.cmd |
startprimary |
|
startsecondary.cmd |
startsecondary |
|
startprimaryasnonleadercmd |
startprimaryasnonleader |
|
startfeedsim.cmd |
startfeedsim |
|
startdequeue.cmd |
startdequeue |
|
common.cmd |
common |
|
basic-common.cmd |
basic-common |
|
custom-common.cmd |
custom-common |
|
StreamBase Studio can run only one server instance at a time, and the HA sample does not make much sense unless both server instances are running. For this reason, you must run the sample in terminal or StreamBase Command Prompt windows as described in the next section.
However, you are encouraged to load each EventFlow application into StreamBase Studio to see how each is assembled.
Note
This sample makes use of environment variables and configuration file global parameters set with the <operator-parameter>
element. See Using Global Parameters in the Authoring Guide for more information.
Global parameters can be used by operators or adapters so that values are not hard-coded in the application. Global parameters that are referenced within an operator must be defined within the configuration file. In the configuration file, you can set the value of a global parameter by means of an environment variable. Putting these two features together allows you to define an global parameter value at runtime.
If you run this sample with the provided scripts, the environment variables are set for you. When experimenting with the sample
in StreamBase Studio, or when running the sample without the provided scripts, see the comments in ha.sbconf
and ha-custom.sbconf
for information on the environment variables that must be set.
The HA sample is shipped with support for StreamBase authentication disabled. To run this sample with StreamBase authentication enabled you must perform the following prerequisite steps:
-
Create an authentication database, usually stored in
$STREAMBASE_HOME/etc
, using the sbuseradmin command. For instructions, see Using StreamBase Server Simple Authentication. -
In the authentication database, create at least one StreamBase username with the SBAdmin role.
-
Set the following environment variables:
AUTH true
AUTH_PASSWORD_FILE Path to the authentication password file created with sbuseradmin. AUTH_ADMINUSER Username assigned the SBAdmin role in the authentication file. AUTH_ADMINPASSWORD Password for the above username. These variables apply to the HA family of samples, and are not StreamBase standard environment variables.
This section shows how to run the samples and suggests some steps to show how the servers, the containers, and the applications work together to ensure high availability. You can, of course, experiment with other steps of your own.
To get each sample up and running you use scripts that do the following:
-
Start the servers.
-
Start a feed simulation to all streams.
-
Start a dequeue from all streams.
The scripts are in one or the other of two directories according to which sample you want to run.
Because StreamBase Studio can run only one server instance at a time, you must run this sample's two servers in terminal or StreamBase Command Prompt windows.
Note
On Windows, be sure to use the StreamBase Command Prompt from the Start menu as described in the Test/Debug Guide, not the default command prompt.
The steps to run the sample are as follows:
-
Set up five terminal windows.
-
Open five terminal windows on UNIX, or five StreamBase Command Prompts on Windows.
-
In each window, navigate to your workspace copy of the sample.
For the basic sample, navigate to the
basic-scripts
folder; for the custom sample, navigate to thecustom-scripts
folder.Note
The steps are the same for each sample. Navigate first to the
basic-scripts
folder, follow the steps for running the sample, then navigate to thecustom-scripts
folder and repeat the steps.
-
-
Start the following processes:
-
In window 1, start the first server with this command:
startprimary
.This script sets environment variables, starts a server listening on port 9900, and loads the high availability application (
ha
) in a container,ha
, on that server. The high availability application designates this server as the leader and startsapp.ssql
.In window 5, run the following command to observe this outcome:
sbc -u :9900 list containers
This command generates the following output:
container _SB_ha container app container ha_control_app container system
-
In window 2, run
startsecondary
.This script performs the same actions as startprimary, except it starts its server on port 9901 and sets its leadership status as nonleader.
-
In window 3, run
startfeedsim
.A feed simulation starts sending input data to both servers. Window 3 shows input, such as the following:
in (a=871) in (a=6419) in (a=2348) in (a=8339) in (a=5707) in (a=909) in (a=8756) in (a=3983) in (a=8748) ...
-
In window 4, run
startdequeue
.The command sent by the
startdequeue
script is a comma-separated list of URIs to address both servers at the same time. After resolving the variables for port number, the command sent bystartdequeue
resolves to the following:sbc -u sb://localhost:9900/app,sb://localhost:9901/app dequeue
Notice the hostname (myhostname) and the StreamBase URI of the responding server is shown for each line, and only one server is responding, namely the server designated as the leader.
out,2555,
myhostname
,sb://localhost:9900/ out,3091,myhostname
,sb://localhost:9900/ out,8482,myhostname
,sb://localhost:9900/ out,520,myhostname
,sb://localhost:9900/ out,101,myhostname
,sb://localhost:9900/ out,3538,myhostname
,sb://localhost:9900/ out,867,myhostname
,sb://localhost:9900/ out,8503,myhostname
,sb://localhost:9900/ out,36,myhostname
,sb://localhost:9900/ ...
-
-
Test the high-availability features of the sample:
-
In window 5, run the following command to determine the status of both servers:
sbc -u :9900,:9901 status --verbose
This command generates a status report line for each server. Notice that the server on port 9900, which is running in window 1, is designated as the leader.
-
In window 5, run the following command to shut down the primary server, the one that is now the leader:
sbadmin -p 9900 shutdown
Notice that the dequeued output stream in window 4 has switched to showing tuples from the server on port 9901:
out,6700,
myhostname
,sb://localhost:9901/ out,7156,myhostname
,sb://localhost:9901/ out,1303,myhostname
,sb://localhost:9901/ out,4798,myhostname
,sb://localhost:9901/ out,2305,myhostname
,sb://localhost:9901/ out,9540,myhostname
,sb://localhost:9901/ ... -
In window 5, run the following command to determine the status of both servers:
sbc -u :9900,:9901 status --verbose
Notice that the server on port 9901, which is running in window 2, is designated as the leader.
-
In window 1, restart the primary server by running the startprimaryasnonleader command. Notice that this server is the nonleader.
-
Shut down the secondary server, the one that is now the leader:
sbadmin -p 9901 shutdown
Notice the output stream in window 4 switches back to the server on port 9900.
-
Check the status again by running the same command as before:
sbc -u :9900,:9901 status --verbose
Now, only one server is running, namely the server on port 9900, and it is the leader.
-
-
To shut down all servers, run the following commands
-
In window 4, press Ctrl+C to stop the dequeuer.
-
In window 3, press Ctrl+C to stop the feed simulation.
-
In window 5, run:
sbadmin -p 9900 shutdown sbadmin -p 9901 shutdown
-
-
Repeat the steps for the custom sample.
For the custom sample navigate to the following directory to run the scripts:
sample_ha/custom-scripts
-
Optionally, experiment with the
haleadershipcontrol.sbapp
to try out different strategies for customized leadership control.
This section describes how to run the hbtester.sbapp
application to learn about the interaction of two instances of the HA Heartbeat adapter and test different scenarios for
using it. This application runs standalone, independent of the other applications in this sample.
Because StreamBase Studio can run only one server instance at a time, you must run this sample's two instances in terminal or StreamBase Command Prompt windows.
The hbtester.sbapp
application can be run as either client or server heartbeat adapter. Run them on separate ports to run them at the same time,
using commands like the following:
-
Set up six terminal windows.
-
Open six terminal windows on UNIX, or six StreamBase Command Prompts on Windows.
-
In each window, navigate to your workspace copy of the sample.
-
-
In window 1, start the first HA Heartbeat instance:
sbd -p 9000 hbtester.sbapp
-
In window 2, start the second instance:
sbd -p 9010 hbtester.sbapp
-
In windows 3 and 4, start dequeuers for each server:
sbc -p 9000 dequeue sbc -p 9010 dequeue
-
In windows 5 and 6, start enqueuers for each server:
sbc -p 9000 enq CommandIn sbc -p 9010 enq CommandIn
-
In windows 5 and 6, experiment with various commands to activate and deactivate the two HA Heartbeat adapters, watching the resulting output messages in windows 3 and 4. For example, try the following sequence:
-
In window 5, enter:
startServer
-
No output is seen yet because no heartbeat clients have yet started. Now in window 6, enter:
startClient
-
Look for output messages in windows 3 and 4 from both servers.
-
In window 6, enter:
stop
-
Look for a
disconnected
message in window 3 and atimeout
message in window 4. -
In window 6, enter:
start
-
Look for the output messages to resume in windows 3 and 4.
-
In windows 5 and 6, stop both server and client heartbeat adapters by entering:
stop
-
Reverse the positions of the server and client heartbeats. In window 6, enter:
startServer
-
In window 5, enter:
startClient
-
Look for the resumption of output messages in windows 3 and 4.
Continue experimenting as you like, paying attention to the content and format of output messages in windows 3 and 4. This will help you design your HA-compliant application to respond to messages from the HA Heartbeat under different circumstances.
-
-
When done:
-
In windows 5 and 6, type Ctrl+C to exit both sbc enq sessions.
-
In another terminal window or StreamBase Command Prompt, type:
sbadmin -p 9000 shutdown sbadmin -p 9010 shutdown
This terminates both servers and both dequeuers, and closes windows 1 through 4.
-
See HA Heartbeat Input Adapter for details on the operation of the HA Heartbeat adapter.
In StreamBase Studio, import this sample with the following steps:
-
From the top menu, select
→ . -
Select the
ha
sample from the High Availability list. -
Click OK.
StreamBase Studio creates a single project containing the sample files. The default project name is sample_ha
.
When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.
Important
Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.
Using the workspace copy of the sample avoids the permission problems that can occur when trying to work with the initially installed location of the sample. The default workspace location for this sample is:
studio-workspace
/sample_ha
See Default Installation Directories for the location of studio-workspace
on your system.
In the default TIBCO StreamBase installation, this sample's files are initially installed in:
streambase-install-dir
/sample/ha
See Default Installation Directories for the default location of studio-workspace
on your system.