Monitoring for Disaster Recovery

FTL exposes the following information to help you monitor operations where disaster recovery is enabled.

  • FTL server's operating mode (usually, primary or DR).

  • The replication status of each persistence service at the DR site. The status tells you if the DR site is actively consuming the tail end of data from the primary site, or if there is a larger gap.

  • Status of persistence stores, durables, and destinations at the DR site, particularly pending message counts.

See the following information for details on how to obtain this information from FTL. Note that when you query FTL, you get only persistence status for persistence services running at that site. So, if you query the active site, you get status for the active persistence services, and if you query the standby site, you get status for the standby persistence services.

FTL Server's Operating Mode

An FTL server can be in one of the following modes:

  • STARTING. The FTL server is initializing and does not know whether it is primary or DR yet. If you are using the simplified: DR model, this can occur in certain corner cases and might indicate that manual intervention is required. See Simplified Disaster Recovery.

  • STANDALONE/PRIMARY. The FTL server is in primary mode, and can have an associated DR site. Administrators can update the FTL realm configuration at this site.

  • DISASTER RECOVERY. The FTL server is receiving FTL realm configuration from an associated primary site. The realm configuration at this site is read-only. Application clients do not connect to this site.

  • SATELLITE. The FTL server is a satellite. It is receiving FTL realm configuration from the primary site. This case only applies if you configure DR for persistence clusters at a satellite site (usually for routing). In this case, the persistence services at a satellite can either be in active mode or standby mode, and there is another satellite site that serves as the corresponding active or standby site. See Disaster Recovery for Routes.

Use one of the following methods to obtain the FTL server's operating mode.

  • FTL server GUI: Open a web browser to the overview page. The UI displays a badge if the FTL server is in STARTING, DISASTER RECOVERY, or SATELLITE mode. See Header.

  • FTL server web API: Retrieve the mode field from the response to GET api/v1/server or GET api/v1/cluster. See FTL Server Status Objects.

  • FTL administration tool: Run the --status command. See FTL Administration Utility.

    Example:

    tibftladmin -ftls <url> --status

Persistence Service Status

A persistence service at the DR site has one of the following states:

  • DR Leader. This persistence service is the leader of the standby set. It receives application data from the active site leader and forwards it to followers at the standby site. This persistence service is actively consuming the tail end of data from the active site (minimal data gap).

  • DR Follower. This persistence service is a follower in the standby set. It receives application data from the standby site leader. This persistence service is actively consuming the tail end of data from the active site (minimal data gap).

  • DR Leader Syncing. This persistence service is the leader of the standby set. It is performing bulk data copy from the active site, possibly due to failure of the WAN link or a server restart at one of the sites. There might be a larger amount of data missing.

  • DR Follower Syncing. This persistence service is a follower in the standby set. It is performing bulk data copy from the active site, possibly due to failure of the WAN link or a server restart at one of the sites. There might be a larger amount of data missing.

  • DR Suspending. A planned failover is in progress. The active site has stopped accepting data, and this persistence service is copying all remaining application data from the active site in anticipation of planned DR failover or failback.

  • DR Suspended. A planned failover is in progress. The active site has stopped accepting data, and this persistence service has received all application data from the active site. This persistence service is ready for planned DR failover or failback.

  • Quorum Offline. This persistence service is offline.

The system is healthy when all persistence services at the standby site have status DR Leader or DR Follower.

Use one of the following methods to obtain persistence service status.

  • FTL server GUI: Navigate to the Persistence Cluster Table or Persistence Cluster Details Page.

  • FTL server web API: For each standby persistence service, retrieve the quorum_state and quorum_state_label fields from the response to GET api/v1/persistence/<cluster>/servers/<server>. See Persistence Status Objects.

  • FTL server web API: As an alternative, retrieve the leader_in_sync field from the response to GET api/v1/persistence/<cluster>/standby.

Remember to issue requests to an FTL server at the standby site to get standby status.

Pending Message Count

It is a good idea to compare pending message counts (and possibly more detailed status of stores and destinations) at the primary and standby sites during normal operation. It is especially important to do so during a planned DR failover or failback, just before activating the DR site. When you activate the DR site and client applications fail over, there is no easy way to recover data at the primary site that was not replicated to the DR site.

Use one of the following methods to obtain pending message counts.

Remember to issue requests to both sites.