================================================
Fault-tolerant LiveView Using TIBCO FTL
================================================

Overview:

This component demonstrates LiveView recovering from a service
interruption (a crash) using TIBCO FTL as the publishing bus. The 
LiveView Orders table is configured to use persistence and it has 
a publisher interface with an FTL message bus subscriber that is 
configured with a durable subscription.

The LiveView recovery protocol is documented in:

LiveView Development Guide -> EventFlow Publish Applications
or
https://docs.tibco.com/pub/streambase_cep/7.5.4/doc/html/lv-devel/lv-publisher.html#lv-devel_persistence

After LiveView service interruption and restart, LiveView will recover
from disk almost all of the rows that were in the Orders table at the
time of the interruption.

It is possible that some in-flight rows were not persisted before the
service interruption occurred. Plus all messages that were published
while the server was unavailable need to be recovered too.

The Orders data is generated in a simple CEP application using a
Feedsim adapter and an FTL publishing adapter. The logic for
this is in MessagePublisher/MessagePublisher.sbapp, while FTL.sbapp
contains mostly the FTL adapter.

The LiveView publisher module is in
LVPublisherApps/LVPublisher.sbapp and it mostly just references
the underlying bus implementation.

FTL Implementation details:

TIBCO FTL documentation can be found in:

  https://docs.tibco.com/

Configuring LiveView/StreamBase to use FTL can be found in:

  https://docs.tibco.com/pub/streambase_cep/x.y.z/doc/html/adaptersguide/embeddedInputTibcoFTL.html
  https://docs.tibco.com/pub/streambase_cep/x.y.z/doc/html/adaptersguide/embeddedOutputTibcoFTL.html

where x.y.z is the StreamBase version being used.

Only rows that are known to be persisted are acknowledged to the FTL
subscriber. Unacknowledged FTL messages will be automatically resent
by FTL when the LiveView server restarts and its FTL subscription is
reestablished.

Typically most of the messages replayed by FTL during recovery will need
to be written to the table, but it is possible there are some messages that
were actually persisted and subsequently recovered by LiveView that were
not acknowledged back to FTL. To avoid republishing these messages the
LiveView recovery logic uses the PublisherID and PublisherSN to detect
when to drop replayed messages that are already recovered.

If re-publishing data is not an issue for your configuration, you do
not have to implement this logic.  "Republishing" identical data to a
table may not be idempotent if you have update field rules defined, or
other statefull configurations involving preprocessors, transforms,
etc.

You must install TIBCO FTL libraries as described in the StreamBase CEP
documentation on TIBCO FTL adapters.  This will include copying a jar file
into the StreamBase installation and setting a load path to the FTL native
libraries.

You will have to have the environment setup to run applications. This
includes having STREAMBASE_HOME/bin and FTL_HOME/bin on your path and
setting the library path to the location you have installed FTL native
libraries:

Defaults on Linux:
  export FTL_HOME=/opt/tibco/ftl/4.1
  export PATH=/opt/tibco/sb-cep/7.6/bin:$FTL_HOME/bin$PATH
  export LD_LIBRARY_PATH=$FTL_HOME/lib/

To make things easier, a StreamBase Command Prompt should be used to 
issue the following commands as it sets up many required system
variables. For those commands which deal with FTL, you should also
run the FTL script which sets up those environment variables specific
to FTL. This is found in TIBCO_HOME\ftl\x.y\samples. See the FTL 
documentation for the name of the script. Also note that all file
paths are relative to where this file is located in a normal sample 
installation.

There is an FTL realm configuration with a durable subscription that
will work with this item. First start the Realm server in the
realm-server directory:

  tibrealmserver

If you have not already configured the realm server, from a different
terminal window in the same directory update the realm with the durable
endpoints/persistence server/etc.:

  tibrealmadmin -rs http://localhost:8080 --updaterealm durable-liveview.json

You only need to do this step once.

You also need a persistence server running - from a different terminal window
in the persist-server directory, do:

  tibstore -n LVPublisherPersistenceServer -rs localhost:8080

See FTL documentation to confirm the realm server is running and the
persistence server is reporting that it is leader.

Start the LiveView Server:

  lv-server run .

In the MessagePublisher directory, start the publisher.

  sbd -f sbd.sbconf MessagePublisher.sbapp

Connect a LiveView desktop to the LiveView server running on port
10087 and confirm you are seeing data. Shutdown the publisher and wait
for the desktop output to stop. Then shutdown the LiveView server.

If you are running on linux - once you have your environments, paths, and
FTL realm configured correctly and have confirmed the simple case of
the publisher and LiveView both running is working correctly there is
a test bash script you can run. Make sure the publisher and LiveView
server are not running.
 
The script starts the publisher and then crashes the LiveView server
(3 times by default). It then compares the data in the LiveView table
with the data that was published. Despite having many service
interruptions, the data should match exactly. The script does this
comparison.  To run this script

bash liveview-bounce.sh

Please note that if you test manually, you must kill the LiveView server
process forcefully. You should use the appropriate tool: taskkill,
Task manager, kill -9, killall, etc.  If you try to simply use ^C (or
its equivalent kill command), the server process will catch the
interrupt and gracefully shutdown, removing the durable subscription
so any messages sent during this time would be lost.

Modifying this component for your uses: 

Note that the interface of record for the users LiveView table is in
MessagePublisher/Orders.sbint. The named schemas in this interface are
imported by the LiveView lvconf files, which in turn are the basis of
the generated interfaces in tables/lv-interfaces. Orders.sbint is
where you would start when making table schema changes for your tables.

Version History:
1.0     Initial release with FTL
1.1     bug fixes to recovery test script