======================================================================
Delayed LiveView Peer-Based Recovery Sample for TCM eFTL Message Bus
======================================================================

Overview:

This sample demonstrates the LiveView delayed peer-based recovery capability 
when using the eFTL message bus service. TIBCO Cloud Messaging (TCM) can 
optionally use eFTL. See the TIBCO Streaming eFTL adapter documentation to 
ensure the eFTL dependencies are properly configured.  

Message buses such as Kafka support replaying older messages based on an 
offset, or sequence number. LiveView recovery with such buses is fairly simple 
and does not need the delayed recovery capability. See the Kafka recovery 
sample for more information.

Buses such as eFTL, FTL, MQTT, and so on, even when configured with 
persistence, do not support playing older messages that were not previously 
received and acknowledged. For buses of this kind, LiveView must use delayed 
peer-based recovery as demonstrated in this sample.

To avoid missing data from eFTL while recovery is in progress, the sample must 
first subscribe to the eFTL service. Only after the subscription is 
successful will the sample start recovering data from peer nodes, and storing 
incoming eFTL data, if any, while peer recovery is ongoing.

During startup, a LiveView node looks for peer nodes in the same cluster and 
starts the delayed recovery process if there is at least one peer node 
available. After peer recovery is complete, retained data from the eFTL 
subscription is sent to the LiveView table and publishing transitions to the 
real-time data coming from the bus.

Non-idempotent LiveView Table Recovery:

During peer recovery it is possible that some small number of messages may 
both be recovered from the peer and delivered by the eFTL subscription.
If the LiveView table is idempotent, this does not matter; the eFTL message 
is harmlessly republished to the table.

However, if the LiveView table is non-idempotent, republishing the same 
message may cause issues. Examples of non-idempotent tables are ones that have 
autoincremented primary keys, or a generated timestamp field. The number of
messages in question is a function of how fast the publish rate is, and how 
long peer recovery takes.

For non-idempotent tables where the primary key fields are all present in the 
data, and rows are known to never be updated, "perfect" recovery is done by 
dropping messages from the retained message bus data that have already been 
recovered from the peer.

For non-idempotent tables where the primary key fields are not all present in 
the data, or row updates are allowed, you must choose a strategy that 
minimizes the impact of an "imperfect" recovery for the individual use case.

While there are a number of special cases described below that can avoid 
recovery defects, ultimately you must choose whether you want the sample to 
either republish a row or drop a message.

Configuration Options:

This sample contains options to guide you through the recovery process, so 
that you can make recovery as accurate as possible - or at least minimize 
recovery defects according to your application design goals. To see the 
options, click the Parameters tab of ItemsSalesPublisher.sbapp, which is 
located in src/main/eventflow under the 
com.tibco.ldm.sample.lv_sample_eftl_delayed_recovery.messagepublisher package.

The options and guidance are described below. Note that non-idempotent 
recovery generally means that all message bus traffic is deferred until peer 
recovery is complete, then the message bus data is delivered.

If primary key fields are available, set the parameter, 
PRIMARY_KEY_AVAILABLE to true (the default).

If the data stream contains primary key fields:

    1. If rows are ever updated: 
        a) If peer recovery never delivered a row with a primary key 
           that was delivered by the message bus. The message bus 
           row should be published and recovery has no defects. 
        b) If both the peer recovery and the message bus delivered the 
           same primary key row, you can specify whether recovery should 
           update the LiveView table with the last message bus row -- or 
           alternatively drop such rows. You can set this using the 
           PUBLISH_IF_DUP parameter. Setting the parameter to true means 
           the sample should drop such rows. If the row is published this 
           may be a duplicate, and any non-idempotent characteristics of the 
           data may be adversely affected. If the row is dropped, updates to 
           existing rows may be lost.

       2. If rows are never updated: 
          Any duplicate message bus rows are dropped and recovery contains no 
          defects.

If there are no fields in the data stream that can be used as exact or 
approximate primary keys:

    1.  Specify whether rows that arrive on the message bus during the 
        recovery process should be dropped or published using the 
        PUBLISH_TCM_DATA_ARRIVE_ON_RECOVERY parameter. True means data 
        arriving during recovery process will be published. If rows are 
        published, then duplicate rows are possible. Otherwise, the rows 
        may be lost during recovery. 

Sample Requirements:

To run this sample, an available eFTL service is required. The sample uses FTL 
6.4 and eFTL 6.4, and assumes the eFTL server is running at 
ws://localhoast:9191 by default.

To set up a local eFTL/FTL service, make sure you have FTL 6.4 and eFTL 6.4 
installed. The FTL installation directory (such as /opt/tibco/ftl/6.4/samples) 
contains a samples directory with a readme.txt that documents how to 
start an eFTL server on the default port. Starting an eFTL server is required 
to run this sample.

For more information about eFTL configurations, see 
    https://docs.tibco.com/products/tibco-eftl-enterprise-edition-6-4-0

Running the Sample:

After starting the eFTL/FTL service, the next step is to publish messages to 
the service:

    Right-click the eFTL.sbapp, located in src/main/eventflow under the 
    com.tibco.ldm.sample.lv_sample_eft_delayed_recovery.messagepublisher 
    package name, and run it as an EventFlow Fragment.

Only one publisher instance is needed.

Once the eFTL service and publisher are running:

    1) Go to Run > Run Configurations and create a new LiveView fragment 
       run configuration. 
    2) Click Browse and select this sample project as the LiveView Project.
    3) Set the LiveView engine port to 10080.
    4) Make sure only engine.conf is selected as the configuration file 
       (by default every file is selected). 
    5) Go to the Node tab and enter peer1$(SeqNum:_) as the node name. 
    6) Click Run.

The above steps start the first node in the cluster. Repeat the steps 
for the peer node, but set the LiveView engine port to 10081 and use
peer2$(SeqNum:_) for the node name.

Expect to see data available on both localhost:10080 and localhost:10081.

Sample Script Details:

The script lv-test-consistency.sh compares data between two servers running 
on ports 10080 and 10081. This is useful as there can be inconsistency at 
the tail end of the data, depending on the timing when the server receives 
eFTL messages. 

The script lv-test-no-duplicates.sh checks whether there is any row that is 
published more than once on the recovered server on port 10081. The 
ItemsSales table contains the field updateCount that increments by 1 every 
time the row is published. The script ensures that no row has an updateCount 
greater than 1.

Modifying the Sample for Your Use Case:

To demonstrate the scenario where updates are possible from the message bus 
when the peer server is still under recovery progress, you can either:

 - edit the ItemsSales.sbfs file, located in src/main/resources, to allow
   a duplicate transactionID.
 - or use UpdateSimulator in ItemsSalesProvider.sbapp. The UpdateSimulator 
   updates an ItemsSales row with transactionID value of 10; you set parameter 
   SIMULATE_UPDATE to true in ItemsSalesProvider.sbapp to start the simulation.

src/main/eventflow/ItemsSalesProvider.sbapp is where the eFTL is subscribed 
and connected to the sample.

src/main/eventflow/ItemsSalesPublisher.sbapp is the publisher application. 
Modify parameters described above as required.

If you need to use your own LiveView tables, make sure to modify named 
schemas in src/main/eventflow/lvinterfaces/ItemsSalesSchemas.sbint, and update 
ItemsSalesPublisher.sbint in the same package.

There is a convenient program ItemsSalesSchemasGenerator.java located in 
src/main/java under the com.tibco.liveview package that generates 
Publisher and Table sbint schemas from the provided LiveView data table.