================================================ Fault-tolerant LiveView Using TIBCO EMS ================================================ Overview: This component demonstrates LiveView recovering from a service interruption (a crash) using TIBCO EMS as the publishing bus. The LiveView Orders table is configured to use persistence and it has a publisher interface with an EMS message bus subscriber that is configured with a durable subscription. The LiveView recovery protocol is documented in: LiveView Development Guide -> EventFlow Publish Applications or https://docs.tibco.com/pub/streambase_cep/x.y.x/doc/html/lv-devel/lv-publisher.html#lv-devel_persistence After LiveView service interruption and restart, LiveView will recover from disk almost all of the rows that were in the Orders table at the time of the interruption. It is possible that some in-flight rows were not persisted before the service interruption occurred. Plus all messages that were published while the server was unavailable need to be recovered too. The Orders data is generated in a simple CEP application using a Feedsim adapter and an EMS publishing adapter. The logic for this is in MessagePublisher/MessagePublisher.sbapp, while EMS.sbapp contains mostly the EMS adapter. The LiveView publisher module is in tables/LVPublisherApps/LVPublisher.sbapp and it really just references the underlying bus implementation. EMS Implementation details: TIBCO EMS documentation can be found in: https://docs.tibco.com/ Configuring LiveView/StreamBase to use EMS can be found in: https://docs.tibco.com/pub/streambase_cep/x.y.z/doc/html/adaptersguide/embeddedEMS.html Only rows that are known to be persisted are acknowledged to the EMS subscriber. Unacknowledged EMS messages will be automatically resent by EMS when the LiveView server restarts and its EMS subscription is reestablished. Typically most of the messages replayed by EMS during recovery will need to be written to the table, but it is possible there are some messages that were actually persisted and subsequently recovered by LiveView that were not acknowledged back to EMS. To avoid republishing these messages the LiveView recovery logic uses the PublisherID and PublisherSN to detect when to drop replayed messages that are already recovered. If re-publishing data is not an issue for your configuration, you do not have to implement this logic. "Republishing" identical data to a table may not be idempotent if you have update field rules defined, or other state-full configurations involving preprocessors, transforms, etc. You must install TIBCO EMS libraries as described in the StreamBase CEP documentation on TIBCO EMS adapters. This will include copying a jar file into the StreamBase installation and setting a load path to the EMS native libraries. You will have to have the environment setup to run applications. The easiest way to do this is to start from a StreamBase command prompt. The sbd.sbconf files for the MessagePublisher and tables are currently configured to use a default Windows installation of EMS 8.1. Modify these to match your installation. Start the EMS server: If you are using an existing server, you will need to configure the various EMS adapters using the EMS/JMS editor. At a minimum, you will need to adjust the server parameters to match your installation. If you are not using an existing EMS installation, you will need to install EMS on the local machine. Then you must start the EMS server. This is done by going to the directory where EMS is installed and, on Windows, running the tibemsd.bat file found in the bin directory. If you simply run tibemsd, the executable will be run and not the batch file which sets up the environment to what is expected by this sample. You need to set EMS_HOME environment variable to point to your EMS installation, for example: export EMS_HOME=/opt/tibco/ems/8.3 Start the LiveView Server: Note that you must start the LiveView server before you start the message publisher. The reason is explained later in this document. lv-server run . After the LiveView server has become ready, you can start the publisher. In the MessagePublisher directory, start the message publisher: sbd -f sbd.sbconf MessagePublisher.sbapp Connect LiveView Desktop or lv-client to the LiveView server running on port 10087 and confirm you are seeing data. Shutdown the publisher and wait for the desktop output to stop. Then shutdown the LiveView server. If you are running on Linux - once you have your environments, paths, and EMS configured correctly and have confirmed the simple case of the publisher and LiveView both running is working correctly there is a test bash script you can run. Make sure the publisher and LiveView server are not running. The script starts the publisher and then crashes the LiveView server (3 times by default). It then compares the data in the LiveView table with the data that was published. Despite having many service interruptions, the data should match exactly. The script does this comparison. To run this script liveview-bounce.sh Please note that if you test manually, you must kill the LiveView server process forcefully. You should use the appropriate tool: taskkill, Task manager, kill -9, killall, etc. If you try to simply use ^C (or its equivalent kill command), the server process will catch the interrupt and gracefully shutdown, removing the durable subscription so any messages sent during this time would be lost. Modifying this component for your uses: Note that the interface of record for the users LiveView table is in Orders.sbint. The named schemas in this interface are imported by the LiveView lvconf files, which in turn are the basis of the generated interfaces in tables/lv-interfaces. Orders.sbint is where you would start when making table schema changes for your tables. Why order matters: Under best practices for a fault tolerant LiveView publisher, an Active-Active setup is used. This means that all messages published to the message bus must go to both instances of the table publisher. Because of this, a fault tolerant EMS instance cannot use a queue which would deliver each message to one of its subscribers, but must use a topic. The topic delivery mode must be persistent and must also must be durable. There is one issue with the EMS/JMS specification - even if the delivery mode is persistent, messages will not be persisted until at least one durable subscription is made. Any messages produced before that will be lost. This is the reason that, in the sample, the table publisher must be started before the message publisher so it can make the durable subscription. In your implementation, you must determine how to do this. Version History: 1.0 Initial release with EMS 1.1 bug fixes to publisher and test script Copyright (c) 2004-2020 TIBCO Software Inc. All rights reserved.