Task Fault Tolerance
Task fault tolerance enables an Engine to continue executing a task even if it logs off of a Broker, so that it does not lose work due to a Broker failure. This means that if an Engine is working on a task, and it logs off the Broker, it does not immediately exit. Rather, it continues to work on that task, while continuing attempts to log in to a Broker with the Service on which it is working. If it does not log back in within a defined time period, it exits. If it does log back in, it notifies the Broker that it is working on the task. If the task is completed, it immediately sends the result; otherwise, it does so upon completion.
Using this feature is only recommended when you have individual tasks that take many hours to finish. For example, if a report runs during the night and some tasks take eight hours to process, task fault tolerance ensures that the eight-hour tasks don’t have to start from the beginning if the Broker fails at 7 AM. Enabling task fault tolerance can diminish the efficiency of the grid, since it redundantly schedules all outstanding tasks. For short tasks, it’s usually more efficient to simply recalculate tasks in the event of a Broker failure.
Consider the following example of task fault tolerance:
| 1. | An Engine and Driver are connected to Broker A. The Driver submitted a Service, and an Engine is working on that Service. |
| 2. | Broker A goes down. |
| 3. | The Driver tries to reconnect with Broker A. The Engine continues working and tries to reconnect to Broker A. |
| 4. | After five minutes, the Driver gives up attempts to connect to Broker A. It connects to Broker B and resubmits outstanding work. |
| 5. | The Service is now on Broker B. The Engine logs in to Broker B and indicates that it is taking that task. If the Engine already finished its work, it immediately writes the task. Otherwise, after it completes its work, it writes its task. |
If another Engine takes the task by the time the original Engine logs in, no attempt is made to cancel the task on the Broker. It is the same as a redundantly rescheduled task.
The situation is similar when an Engine logs into a Failover Broker and works on a task. When the Driver switches back to the Primary Broker, the Engine logs off the Failover Broker and reconnects to the Primary Broker. The task is not canceled.
To enable task fault tolerance, go to Admin > System Admin > Manager Configuration > Engines and Clients, and change the value of Engine Timeout Minutes. Make the Engine timeout longer than the Driver’s timeout, which is the value of DSBrokerTimeout set in the driver.properties file (five minutes by default.) Note that changes to this value take effect at the next Engine Login.
To use task fault tolerance, another Broker must be available for failover, and the Client running the session must fail over to the Broker and resubmit its session.
No attempt is made upon login of the Engine running a fault-tolerant task to cancel that same task if it has already been taken by another Engine.