Advanced Persistence Cluster Configuration

To configure Clusters using a GUI grid, ensure that you are in the Peer-to-Peer UI mode, navigate to the Clusters page through the left menu, and turn on edit mode. For a simplified Cluster Configuration workflow, switch to the Destinations UI mode. See Persistence Cluster Configuration for more information.

For background information, see Persistence Services and Clusters in TIBCO FTL® - Enterprise Edition Administration.

For consequences of modifying this configuration, see Persistence Modifications: Store and Durable and Quorum Behaviors in TIBCO FTL® - Enterprise Edition Administration.

The Cluster Grid has the following Levels: Cluster, Service Set, and Service.

Cluster Level

Column	Description
Cluster	Required. Name of the persistence cluster. Cluster names must be unique. All names are limited to maximum length of 256 characters.
Last Modified	This timestamp indicates the date and time of the most recent change to this persistence cluster definition.
Disk Persistence Mode	Select the disk persistence mode. This option applies to replicated stores only. If you turn on disk persistence but have not turned on swapping, the messages are still held in memory. Once you turn on swapping, the messages are swapped out to disk and freed from memory. None Disable disk persistence. sync The send-message call returns once the message has been written persistently to a majority of disks. async The send-message call returns once the OS has buffered the write of the message. A tail end of data can be lost if a majority of server hosts crash simultaneously.
DR Enabled	Disaster Recovery Replication In the GUI, select this checkbox to enable replication to the disaster recovery service set. In the web API, set this attribute to `true` to enable replication to the disaster recovery set.
DR Protocol	If you enable disaster recovery replication, the services in the cluster use this transport protocol to establish a communication bus for that purpose. You can set other parameters of the transport for each service in the cluster.
Primary Set	In the GUI, select one service set to be primary. In the web API, set this attribute to the name of the primary service set. The opposite service set is the standby set.

Service Set Level

Each persistence cluster can contain either one or two sets of services. To configure Disaster Recovery, configure two service sets; otherwise, only configure one service set.

Column	Description
Set Name	Name of the service set. You may use the default set names or supply other names. Set names must be unique within the cluster. All names are limited to maximum length of 256 characters.
Cluster Set Protocol	Persistence services with a cluster's service set communicate with one another using this transport. Select the transport protocol from the drop-down menu in this column. As needed, set the host, subnet, and port parameters in service-level columns. Set other parameters in the service details panel. Notice that all the services in a service set use the same cluster protocol, though individual services can supply different transport parameters. For connection-oriented protocols, such as static TCP, the realm service automatically arranges the listening ends and connecting ends.
Client Protocol	Client applications communicate with persistence services in the cluster using the client transport. Select the client transport protocol from the drop-down menu in this column. As needed, set the host, subnet, and port parameters in service-level columns. Set other parameters in the service details panel. Notice that all the services in a cluster use the same client protocol, though individual services can supply different transport parameters. For connection-oriented protocols, such as static TCP, the persistence service always is the listening end, and clients the connecting end.

Column

Description

Set Name

Name of the service set.

You may use the default set names or supply other names.

Set names must be unique within the cluster.

All names are limited to maximum length of 256 characters.

Cluster Set Protocol

Persistence services with a cluster's service set communicate with one another using this transport.

Select the transport protocol from the drop-down menu in this column. As needed, set the host, subnet, and port parameters in service-level columns. Set other parameters in the service details panel.

Notice that all the services in a service set use the same cluster protocol, though individual services can supply different transport parameters.

For connection-oriented protocols, such as static TCP, the realm service automatically arranges the listening ends and connecting ends.

Client Protocol

Client applications communicate with persistence services in the cluster using the client transport.

Select the client transport protocol from the drop-down menu in this column. As needed, set the host, subnet, and port parameters in service-level columns. Set other parameters in the service details panel.

Notice that all the services in a cluster use the same client protocol, though individual services can supply different transport parameters.

For connection-oriented protocols, such as static TCP, the persistence service always is the listening end, and clients the connecting end.

Service Level

If the cluster enables the disaster recovery feature, then each set must contain equivalent services. For background information, see Disaster Recovery in TIBCO FTL® - Enterprise Edition Administration.

Column	JSON Attribute	Description
Service Name	`name`	Required. Name of the persistence service. Persistence service names must be globally unique. All names are limited to a maximum length of 256 characters.
Weight	`weight`	Optional. This value must be an integer in the range [1,10]. Larger values indicate greater preference that the service becomes the leader. When absent, the default weight is 10.
Cluster Host/Subnet Cluster Port		Persistence services inherit the cluster transport protocol from the service set. For connection-oriented transports, set the host name and port for establishing a connection (both are required). For dynamic TCP transports, you may set a subnet mask and port (both are optional).
Client Host/Subnet Client Port		Persistence services inherit the client transport protocol from the service set. For connection-oriented transports, set the host name and port for establishing a connection (both are required). For dynamic TCP transports, you may set a subnet mask and port (both are optional).
DR Host DR Port		Persistence services inherit the disaster recovery transport protocol from the cluster. Set the host name and port for establishing a connection (both are required).

Cluster Details Panel

You can adjust the persistence heartbeat and timeout parameters, which detect persistence service availability. For best results, use the default values, except to resolve specific issues or as directed by TIBCO personnel.

GUI Parameter	JSON Attribute	Description
Persistence Heartbeat (Server to the Client)	`client_pserver_heartbeat`	The leader persistence service sends heartbeats to its clients at this interval, in seconds. The default is 2 seconds.
Timeout Interval (Sever to Client)	`client_timeout_pserver`	When the leader’s heartbeat is silent for this interval, in seconds, its clients seek to connect to a new leader from among the other services in the cluster. The default is 5 seconds. When a client's heartbeat is silent for this interval, the persistence service clears all state associated with the client.
Persistence Heartbeat (Server to Server)	`pserver_pserver_heartbeat`	Persistence services in the cluster exchange heartbeats at this interval, in seconds. The default is 0.5 seconds.
Timeout Interval (Server to Server)	`pserver_timeout_pserver`	When the heartbeat of any service in the cluster is silent for this interval, in seconds, the remaining services attempt to form a new quorum. The default is 3 seconds.
Persistence Heartbeat (Cluster to Cluster)	`inter_cluster_heartbeat`	The leader persistence services of different persistence clusters in a forwarding zone exchange heartbeats at this interval, in seconds. The default is 2 seconds.
Timeout Interval (Cluster to Cluster)	`inter_cluster_timeout`	When the heartbeat from a leader persistence service in a second cluster is silent for this interval, in seconds, the leader of the first cluster will attempt to reconnect. The default is 5 seconds.

Disk Settings

For discussion on these features, see Persistence Concepts and Disk-Based Persistence in TIBCO FTL® - Enterprise Edition Administration.

GUI Parameter	JSON Attribute	Description
Message Swapping	`disk_swap`	When enabled, message data is swapped to disk. This allows the persistence service to store more message data than available memory. This setting affects both persistent (replicated) and non-persistent (non-replicated) messages. Note that, if the indexes on disk feature is enabled, persistent messages are always swapped to disk, regardless of this setting. Changing this setting requires a restart of all persistence services in the cluster.
Disk Persistence Mode	`disk_persistence`	Select the disk persistence mode. Only persistent (replicated) messages are persisted on disk. None Disable disk persistence. sync The send-message call returns once the message has been written persistently to a majority of disks. async The send-message call returns once the OS has buffered the write of the message. A tail end of data can be lost if a majority of server hosts crash simultaneously. Changing this setting requires a restart of all persistence services in the cluster.
Automatic compaction	`disk_compact`	Set to enable or disable automatic disk compaction. This setting applies to both non-persistent messages (if swapping is enabled) and persistent messages. Note that, if the indexes on disk feature is enabled, automatic compaction is always enabled for persistent messages, regardless of this setting. Changing the setting does not require a restart.
Compact When In-Use Ratio Below	`disk_compact_settings.min_disk_inuse_ratio`	Set a value between 0 and 1, inclusive. The persistence service attempts to keep the ratio of disk inuse size to disk allocated size at or above the specified value. If the ratio of in-use to allocated size falls below this value, the persistence service can start a compaction. Automatic compaction does not occur if the disk inuse size is extremely small. Automatic compaction must be enabled for this value to take effect. Note that, if the indexes on disk feature is enabled, this threshold is ignored for persistent messages. Automatic compaction is always enabled for persistent messages. Changing the setting does not require a restart.
Indexes on Disk	`disk_index`	When enabled, message data, indexes, and metadata for persistent messages are stored on disk. No memory is consumed to store persistent messages. Also, when the persistence service restarts, it can recover from disk immediately (without having to read message metadata from disk). Changing this setting requires a restart of all persistence services in the cluster.

Cluster Limits and Force Quorum Settings

GUI Parameter JSON Attribute Description

GUI Parameter	JSON Attribute	Description
Force Quorum Delay	`force_quorum_delay`	Automatically force a quorum after this delay, in seconds. Zero is a special value indicating that automatic force formation is disabled. This setting can be used to ensure that the quorum always forms without administrator intervention. A majority of non-empty servers is usually enough to form a quorum, however in some situations (such as DR activation or recovery from a backup file), the quorum can't form unless all members are present. If some servers are still missing, this option forces the quorum to form automatically after the specified delay. Administrators may still manually force a quorum if this setting is configured. See Quorum Behaviors and Before Forcing a Quorum in TIBCO FTL® - Enterprise Edition Administration.
Byte Limit	`bytelimit`	This parameter limits the volume of message data in the cluster. If publishing another message would exceed this limit, the cluster rejects that message. The store may grow until it exhausts available memory or disk of any one of its persistence services. Zero is a special value, indicating no limit. The store may grow until it exhausts available memory of any one of its services. When messages are stored on disk, message metadata is still stored in memory. Use this setting to constrain memory for message metadata. See Size Units Reference in TIBCO FTL® - Enterprise Edition Administration.
Message Limit	`message_limit`	This parameter limits the number of messages that can be held by the cluster. When this limit is reached, the client receives an exception on `tibPublisher_Send` or `tibPublisher_SendMessages` (when `cluster_confirm_send` mode is set on the cluster). However, the client library retries the sends. To have the exception presented to the user, the client application must specify a finite persistence retry duration. Note that even when messages are stored on disk, message metadata is still stored in memory. Hence, use this setting to constrain memory use. Zero is the default value indicating no limit. The cluster may grow until it exhausts available memory or disk of any one of its persistence services.

Force Quorum Delay

force_quorum_delay

Automatically force a quorum after this delay, in seconds. Zero is a special value indicating that automatic force formation is disabled.

This setting can be used to ensure that the quorum always forms without administrator intervention. A majority of non-empty servers is usually enough to form a quorum, however in some situations (such as DR activation or recovery from a backup file), the quorum can't form unless all members are present. If some servers are still missing, this option forces the quorum to form automatically after the specified delay.

Administrators may still manually force a quorum if this setting is configured. See Quorum Behaviors and Before Forcing a Quorum in TIBCO FTL® - Enterprise Edition Administration.

Byte Limit

bytelimit

This parameter limits the volume of message data in the cluster. If publishing another message would exceed this limit, the cluster rejects that message.

The store may grow until it exhausts available memory or disk of any one of its persistence services.

Zero is a special value, indicating no limit. The store may grow until it exhausts available memory of any one of its services.

When messages are stored on disk, message metadata is still stored in memory. Use this setting to constrain memory for message metadata.

See Size Units Reference in TIBCO FTL® - Enterprise Edition Administration.

Message Limit

message_limit

This parameter limits the number of messages that can be held by the cluster. When this limit is reached, the client receives an exception on tibPublisher_Send or tibPublisher_SendMessages (when cluster_confirm_send mode is set on the cluster). However, the client library retries the sends. To have the exception presented to the user, the client application must specify a finite persistence retry duration.

Note that even when messages are stored on disk, message metadata is still stored in memory. Hence, use this setting to constrain memory use.

Zero is the default value indicating no limit. The cluster may grow until it exhausts available memory or disk of any one of its persistence services.

Externally Reachable Addresses

When using persistence with zones of clusters that use an auto inter-cluster transport to address the cluster (or to address a load balancer's interface), use this field to identify one or more ports for the core servers of this cluster (or the port for the load balancer).

GUI Parameter	Description
Host	Enter either an IP address or a resolvable hostname.
Port	Enter the port number to use to access this host.

Persistence Service Details Panel

The Persistence Service details panel presents the details of a persistence service definition. In edit mode, you can modify the definition.

From the left menu, select Clusters, select the three dots above a cluster's Server Name, select View Details.

The title of the panel indicates both the cluster and service names: Cluster / Service.

For background information, see Persistence Services and Clusters in TIBCO FTL® - Enterprise Edition Administration.

For consequences of modifying this configuration, see Persistence Modifications: Store and Durable in TIBCO FTL® - Enterprise Edition Administration.

Service Parameters

GUI Parameter	Description
Server Weight	Optional. This value must be an integer in the range [1,10]. Larger values indicate greater preference that the service becomes the leader. When absent, the default weight is 10.

GUI Parameter

Description

Server Weight

Optional.

This value must be an integer in the range [1,10]. Larger values indicate greater preference that the service becomes the leader.

When absent, the default weight is 10.

Transports

Each persistence service definition must configure a cluster transport and a client transport.

You can also configure an alternate client transport, and a disaster recovery transport, each of which enables optional functionality.

Supply the transport parameters as needed. The available parameters depend on the transport protocol. In general, only a subset of parameters for that transport protocol are available, as the FTL server automatically supplies values for the remainder.

For transport parameter details, see Transport Concepts, and Transport Details Panel in TIBCO FTL® - Enterprise Edition Administration.

For background information, see Persistence Service Transports in TIBCO FTL® - Enterprise Edition Administration.

GUI Parameters Grouping	Description
Client Transport	Client applications communicate with persistence services in the cluster using this transport. Persistence services inherit the client transport protocol from the service set. Select the transport Protocol in the Client Protocol column in the clusters grid. Set other parameters in this section: The Backlog Buffer Size of the persistence client transport limits the number of messages that can be delivered at once. (Messages may also be limited by the prefetch count parameter. See Durable Prefetch Count in TIBCO FTL® - Enterprise Edition Administration.) Receive Spin Limit is an advanced function. See Receive Spin Limit in TIBCO FTL® - Enterprise Edition Administration. See also Client Transport and Non-Blocking Send in TIBCO FTL® - Enterprise Edition Administration.
Cluster Transport	Persistence services with the cluster communicate with one another using this transport. Persistence services inherit the cluster transport protocol from the service set. Select the transport protocol in the Cluster Protocol column of the clusters grid. Set other parameters in this section.
Alternate Client Transport	Optional. When defined, clients on the associated hosts communicate with this persistence service using the alternate transport. All other clients use the regular client transport defined for the cluster. To define an alternate client transport for clients on specific host computers, supply a transport protocol and other transport parameters. Add hosts to the list of Associated Client Host Names. For background information, see Alternate Client Transport in TIBCO FTL® - Enterprise Edition Administration. See also Client Transport and Non-Blocking Send in TIBCO FTL® - Enterprise Edition Administration.
Disaster Recovery Transport	Optional. When two persistence services need to communicate across the WAN for disaster recovery replication, they establish a transport using this definition. Persistence services inherit the disaster recovery transport protocol from the cluster. Select the transport protocol in the DR Protocol column of the clusters grid. Set other parameters in this section. For background information, see Disaster Recovery in TIBCO FTL® - Enterprise Edition Administration.
Zone Transport	Optional. When two persistence services need to communicate across the WAN to forward wide-area stores, they establish a transport bus using parameter values in this definition. You may modify the default parameter values in this section.

Did you find this helpful?

Yes No