Disk-Based Persistence

Disk-based persistence is enabled by default for new users of FTL. Message swapping is enabled by default, and the indexes on disk feature is enabled by default.

Use disk-based persistence to reduce memory costs or improve durability of message data. For an introduction, see Storage and Indexes on Disk.

Consider the following choices (see Cluster Details Panel):

• Disk persistence can be synchronous or asynchronous.

• You might optionally enable the indexes on disk feature. See Using the Indexes on Disk Feature.

• You might optionally enable message swapping.

  • If the indexes on disk feature is enabled, persistent (replicated) messages and their metadata are always swapped to disk, regardless of the message swapping setting. The message swapping setting only affects non-persistent (non-replicated) messages.

  • If the indexes on disk feature is not enabled, the message swapping setting affects both persistent (replicated) and non-persistent (non-replicated) messages.

• You might optionally enable automatic compaction.

  • If the indexes on disk feature is enabled, this setting affects only the temporary swap file for non-persistent (non-replicated) messages.

  • If the indexes on disk feature is disabled, this setting affects both persistent (replicated) and non-persistent (non-replicated) messages.

You must also specify a data directory in the FTL server yaml configuration file for each persistence service that uses disk persistence. The persistence service stores all data in the specified data directory. See FTL Server and Services.

When disk persistence is enabled, you can instruct FTL server to create a backup at any time, even when FTL clients are running. See Disk Persistence Backup and Restore. You must not directly copy the data directory, or any files in the data directory, when the persistence service is running. The resulting copy might represent an inconsistent or incomplete state.

As messages are consumed, the persistence service can compact the data directory to reduce the amount of disk allocated by the persistence service. Compaction can occur when the persistence service is online or offline.

• If the persistence service is online, you might trigger compactions manually through the admin tool, or configure the persistence cluster to perform compaction automatically without intervention from the administrator. Automatic compaction is enabled by default. See Compact Disk Persistence Files with Persistence Service Online.

• If the persistence service is offline, you might use the admin tool to compact the data directory. See Compact Disk Persistence Files with Persistence Service Offline for details.

By default, the persistence service stops accepting published messages if the filesystem containing the data directory is nearly full. See Persistence Service Disk Capacity.