Restarting a Persistence Cluster with Saved State

Administrators can restart a persistence cluster using previously saved state.

Prerequisites

The cluster and all the services in it must be suspended.

You have saved the state of the suspended services

If you saved the state of all the persistence services in the cluster, the recovery is quicker because the services do not need as much time to synchronize.

Nonetheless, if you saved the state of only one persistence service, then the entire cluster can still recover by synchronizing.

Procedure

  1. Determine the order in which you will restart the services.
    If you saved the state of only one service, you must begin with that service.

    If you saved the state of all the services, you may begin with any service.

    After selecting the first server, you may order the others according to your convenience.

  2. Ensure that each relevant state file is in the correct location.
    The default location is:
    data_dir/service_name.state

    If you specified a non-default location using the persistence load parameter in the FTL configuration file, then ensure that the state file is in that location.

  3. Restart the suspended services, one at a time, in the order you determined.
    1. Stop the service.
      Send a stop request to the FTL server that provides the persistence service.
      curl -X POST http://ftl_svr_host:ftl_svr_port/api/v1/server -d '{"cmd":"shutdown"}'
      The FTL server shuts down.
    2. Restart the FTL server.
    3. Wait for the service to report that it has status Running.
    4. Repeat this step for the next suspended service.
  4. Verify cluster status in the persistence clusters status table, and service state in the services list.
    Persistence functionality resumes when the services can form a quorum. For details, see Quorum Conditions General Rule.

    To prevent reloading out-of-date state files at a subsequent restart, the persistence service moves the state file automatically after successfully loading the state file.