Destination Types

Topics

Topics can be used to implement a publish-and-subscribe messaging system.

In a publish-and-subscribe message system, producers address messages to a topic. In this model, the producer is known as a publisher and the consumer is known as a subscriber.

Many publishers can publish to the same topic, and a message from a single publisher can be received by many subscribers. Subscribers subscribe to topics, and all messages published to the topic are received by all subscribers to the topic. This type of message protocol is also known as broadcast messaging because messages are sent over the network and received by all interested subscribers, similar to how radio or television signals are broadcast and received.

Subscribers

In FTL, when an application program creates a subscriber on a topic, FTL will match the application’s destination with a configured destination. See Destination Matching and Wildcards for a description of the destination matching algorithm. If no configured destination matches, the subscribe call will fail.

FTL then retrieves the persistence cluster and store from the configured destination. Recall that a configured destination is always associated with a single persistence store and cluster. (A persistence cluster is a group of persistence services, running in FTL servers, that cooperate to manage a persistence store. A persistence store is a group of messages that share a configuration, including parameters for replication and delivery assurance.)

The persistence cluster for that topic then creates a subscription on behalf of the client program. On success, the subscribe call will return.

Durables

All topic subscriptions are durable subscriptions. This means that the subscriber’s interest for messages on this topic will be maintained even if the subscriber is disconnected or offline. If the subscriber is not available, the persistence cluster will store messages sent to that topic, according to the configuration of the persistence store, and deliver them to the subscriber at the next opportunity.

The durable subscription is identified by the durable name supplied by the application program. If the subscriber restarts or reconnects, it will use the durable name to resume consuming messages on its topic. In a given persistence store, no two durables can have the same name, even if they represent subscriptions on different topics.

There can be multiple durable subscriptions (using different durable names) on the same topic. A message is stored until acknowledged by all subscribers.

Note that a few things can modify the above behavior:

Use of the durable_ttl destination property. See Destination Details Panel.
If the application program does not supply a durable name, the persistence cluster will delete the subscription when the subscriber disconnects, or when the application program closes the subscriber.
Client programs may issue the unsubscribe call once all subscribers are closed, which instructs the persistence cluster to destroy the durable subscription identified by the durable name.

When the persistence cluster destroys a subscription, all pending messages for that subscription are discarded.

To avoid accumulation of messages, when a subscription is no longer needed, client programs should issue the unsubscribe call. Administrators can also delete durables on any topic.

Durable Types

When subscribing on a topic, the application program must specify the type of the subscription.

Standard: The durable subscription is associated with one subscriber. All messages sent to the topic are delivered to all standard subscriptions on the topic. Creating a second subscriber with the same durable name will cause the first subscriber to be closed, and messages will be delivered to the second subscriber.
- If you want to have redundant subscribers on a standard durable, consider using the application group feature. See Groups of Applications.
Shared: The durable subscription is associated with one or more subscribers. All messages sent to the topic are delivered to one of the subscribers. Shared subscriptions allow multiple client programs to parallelize the work of consuming messages on a topic.
Lastvalue: The durable subscription is associated with one or more subscribers. When a subscriber starts, it receives the last message sent on its topic (rather than all messages). Subsequent messages published to the topic are delivered to each online subscriber on that topic. Lastvalue subscriptions allow subscribers to initialize to the most recent state for a topic, and then receive updates for that topic.
- Unlike standard and shared subscriptions, for lastvalue subscriptions the persistence cluster stores only the last message sent to the topic. Publishing a message on the topic will cause the previous message to be replaced.
- When an application program creates a lastvalue subscriber on a topic, the persistence cluster will store the last message sent to that topic or to any siblings of that topic (foo.1, foo.2, foo.3, etc). This is to allow applications to maintain lastvalue state when the set of topics is not known ahead of time (e.g., ticker symbols).
- For example, if a lastvalue subscriber is created on foo.1, then a second lastvalue subscriber, using the same durable name, created on foo.2 would receive the last message published to foo.2. However, messages published to bar or foo.1.2 would not be stored.

It is possible to create standard, shared, and lastvalue subscriptions (using different durable names) on the same topic.

Message Acknowledgment

Lastvalue subscribers do not acknowledge messages. The most recent message is replaced whenever a new message is published to the topic.

Standard and shared subscribers do acknowledge messages. A message is not deleted by the persistence cluster until it has been acknowledged for all subscriptions on the topic.

If the application program specifies explicit acknowledgment mode, then the application program must issue an acknowledge call for each message.
Otherwise, FTL will automatically issue the acknowledge call once the subscriber’s dispatch callback returns, for all messages processed by the callback.

Client programs may specify the acknowledgment mode.

Sync acks: Acknowledgment calls are synchronous. The call does not return until the persistence cluster confirms that it has received the acknowledgment. This type of acknowledgment minimizes duplicates.
Async acks: Acknowledgment calls are asynchronous. FTL will send the acknowledgment in the background on a timer. Duplicates can occur if the application program fails before the acknowledgment is sent.

Publishers

When an application program publishes a message to a topic, FTL will identify the configured destination, persistence store, and persistence cluster in the same way as it does for subscribers.

FTL will send the message to the indicated persistence cluster. If there are subscriptions on the topic, the message will be delivered to those subscribers. The message will be stored until acknowledged by all subscribers.

If there are no subscriptions on the topic, the message will be discarded.

The persistence store determines whether messages are replicated. A persistence store can be replicated (“persistent”), in which case all members of a persistence cluster store a copy of the message, or non-replicated (“non-persistent”), in which case the message is stored only in the memory of the persistence cluster leader.

The persistence store determines the degree of delivery assurance. If the persistence store is configured with publisher mode store_confirm_send, the send call will not return until the persistence cluster has stored the message (or discarded the message in the case of no interest). If the persistence store is configured with publisher mode store_send_noconfirm, the send call will return immediately, regardless of whether the persistence cluster received the message.

Queues

Queues can be used to implement a point-to-point messaging system.

In a point-to-point messaging system, there is one producer and one consumer per message. This style of messaging uses a queue to store messages until they are received. The message producer sends the message to the queue; the message consumer retrieves messages from the queue and sends acknowledgment that the message was received.

More than one producer can send messages to the same queue, and more than one consumer can retrieve messages from the same queue.

Queues are named by destination. There can only be one queue per queue destination.

Subscribers

In FTL, when an application program creates a subscriber on a queue, FTL will match the application’s destination with a configured destination. See [Destination Matching and Wildcards] for a description of the destination matching algorithm. If no configured destination matches, the subscribe call will fail.

If the queue does not already exist, the persistence cluster will create it. On success, the subscribe call returns.

Multiple subscribers may be created on the same queue. In this case messages are divided between the subscribers for processing.

Message Acknowledgment

Queue subscribers acknowledge messages. A message is not deleted by the persistence cluster until it has been acknowledged by a subscriber.

If the application program specifies explicit acknowledgment mode, then the application program must issue an acknowledge call for each message.
Otherwise, FTL will automatically issue the acknowledge call once the subscriber’s dispatch callback returns, for all messages processed by the callback.

Client programs may specify the acknowledgment mode.

Sync acks: Acknowledgment calls are synchronous. The call does not return until the persistence cluster confirms that it has received the acknowledgment. This type of acknowledgment minimizes duplicates.
Async acks: Acknowledgment calls are asynchronous. FTL will send the acknowledgment in the background on a timer. Duplicates can occur if the application program fails before the acknowledgment is sent.

Publishers

When an application program publishes a message to a queue, FTL will identify the configured destination, persistence store, and persistence cluster in the same way as it does for subscribers.

FTL will send the message to the indicated persistence cluster. If the queue does not exist, it will be created by the persistence cluster. The message will be stored even if subscribers have never existed for the queue.

Queue Deletion

Only administrators can delete queues. Deleting a queue will discard all pending messages in the queue.

Maps

Maps can be used for long-term storage of data that is meant to be read repeatedly, rather than consumed once.

For example, if there is state associated with an application program, that state could be stored in a map. Client programs could read that state on startup or at any convenient time.

Maps store key-value pairs. The key is a string determined by the application. The value is an FTL message, containing whatever fields the application needs. Applications may get, set, or delete keys from the map.

In the map case, the destination is just a name for the map. There can only be one map per map destination.

Map Creation

In FTL, when an application program creates a map object on a map destination, FTL will match the application’s destination with a configured destination. See Destination Matching and Wildcards for a description of the destination matching algorithm. If no configured destination matches, the subscribe call will fail.

If the map does not already exist, the persistence cluster will create it. On success, the map creation call returns.

Multiple client programs can create map objects on the same map destination. They all refer to the same map at the persistence cluster, with the same set of key-value pairs.

Map Keys

Calls to get, put, or delete keys are always synchronous. The call does not return until the persistence cluster either responds with the value (in case of get), or confirms that the key has been updated or deleted.

The persistence store determines whether key-value pairs are replicated. A persistence store can be replicated (“persistent”), in which case all members of a persistence cluster store a copy of the key-value pair, or non-replicated (“non-persistent”), in which case the key-value pair is stored only in the memory of the persistence cluster leader.

Map Deletion

Client programs may issue a call to delete a map. This causes the persistence services to destroy all key-value pairs in the map.

Administrators may also delete maps.

Did you find this helpful?

Yes No