Performance Considerations for Persistence

Performance is driven first and foremost by the degree of delivery assurance and durability required by your application. For example, if messages must be published synchronously, replicated, written synchronously to disk, and acknowledged synchronously, your producers and consumers are highly sensitive to the latency of your network and disk. To a degree, dedicating additional CPU and memory or provisioning faster disks can improve performance. Larger performance gains might be possible if synchronous behavior can be relaxed.

Conversely, your goal might be to minimize CPU or memory usage, or something else entirely. Use the following as a checklist for tradeoffs that you might want to make.

At a high level, these are the steps in the data path for broker-based messaging:

  • When an application publishes a message, the message is sent (over TCP) to the FTL server. See Performance: TCP Listen Port and Performance: TCP Receive Spin Limit.

  • The application might choose to publish the message synchronously (wait for an acknowledgment from FTL server), or return immediately from the send call (without confirmation from the server). See Performance: Publisher Send Mode.

  • The application might choose to batch multiple messages in a single send call (for greater performance) or send messages one-by-one (FTL server must confirm each message before the application moves on to the next one). See Performance: Publisher Send Policy.

    • When sending messages one-by-one, there is an option to allow the FTL library to batch these messages automatically. In this case, FTL server no longer confirms each message individually; instead, the application defines a maximum “window” of outstanding messages, and blocks when the window size is reached. Applications enable this behavior by setting the “send policy” to “non-inline”.

  • FTL server can treat the message as persistent or non-persistent. Non-persistent messages are lost when the leader of the persistence cluster restarts. If the message is persistent, it is replicated to all members of the persistence cluster. See Performance: Persistent and Non-Persistent Messages.

  • If the message is persistent, it might be stored in memory or written to disk for greater durability. See Performance: Disk Persistence and Performance: Indexes on Disk.

  • Once the message is replicated and stored (if needed), FTL server sends a confirmation to the publisher (if needed).

  • Both persistent and non-persistent messages might optionally be swapped to disk to reduce server memory usage. See Performance: Message Swapping.

  • FTL server delivers the message to interested consumers.

  • When a consumer receives a message from FTL server, it executes a message callback provided by the application (your code).

  • When your code is finished with the message, the message must be acknowledged to FTL server (so that FTL server can delete the message from storage). You may choose to acknowledge messages explicitly in your code, or allow the FTL library to acknowledge messages automatically after your dispatch callback returns.

  • Messages might be acknowledged synchronously (the application blocks until FTL server has persisted the acknowledgment) or asynchronously (the acknowledgment is transmitted in the background). See Performance: Subscriber Acknowledgments.