Configuration

Configuration
Prev	Chapter 6. Distribution and High Availability	Next

Distribution configuration is described in this section. Distribution configuration is used for:

high availability cluster behavior
distributed transaction behavior
network information
dynamic and static discovery parameters

Distribution configuration has a configuration type of distribution.

Distribution services are enabled when there is an active distribution configuration on a node and an application JVM executing in which to host distribution. Distribution services are disabled when there is no active distribution configuration on a node, or there are no active application JVMs running.

Distribution configuration defines a nested configuration block named Distribution. The distribution configuration block contains these nested configuration blocks:

HighAvailability - high availability configuration
Transaction - distributed transaction configuration
Transport - network configuration
DynamicDiscovery - dynamic service discovery configuration
StaticDiscovery - static service discovery configuration. Contains a separate configuration block for each remote node being defined.

Example 6.1, “Distribution configuration” shows how the distribution configuration and configuration blocks are used.

Example 6.1. Distribution configuration

//
//    Define a distributed configuration named sampleDistribution
//    This is version 1.0 of this configuration
//
configuration "sampleDistribution" version "1.0" type "distribution"
{
     //
     //    Required configuration block name space for distribution configuration
     //
     configure switchadmin
     {
          //
          //    Distribution configuration block
          //
          configure Distribution
          {
               //
               //    High availability configuration
               //
               HighAvailability
               {
                    ...
               };

               //
               //    Distributed transation configuration
               //
               Transaction
               {
                    ...
               };

               //
               //    Distribution transport configuration
               //
               Transport
               {
                    ...
               };
 
               //
               //    Dynamic service discovery configuration
               //
               DynamicDiscovery
               {
                    ...
               };
  
               //
               //    Static service discovery configuration
               //
               configure StaticDiscovery
               {
                    //
                    //     Remote node configuration
                    //
                    RemoteNode
                    {
                         ...
                    };
                    ...
               };
          };
     };
};

The configuration values supported by the distribution configuration blocks are summarized in the tables below.

Table 6.2, “High availability configuration” defines the high availability configuration block values.

Table 6.2. High availability configuration

Name	Type	Description
`nodeQuorum`	Enumeration - `Enable` or `Disable`.	Control node quorum behavior. Default value is `Disable`.
`minimumNumberQuorumNodes`	Integer	Number of nodes required for a quorum. This value is ignored if `nodeQuorum` is set to `Disable`. `minimumNumberQuorumNodes` must be set to 0 if `nodeQuorumPercentage` is set to a non-zero value. Default value is 0. See the section called “Node quorum management” for more details on the use of this configuration value.
`nodeQuorumPercentage`	Integer	Percentage of votes required for a quorum, including the votes for this node. This value is ignored if `nodeQuorum` is set to `Disable`. `nodeQuorumPercentage` must be set to 0 if `minimumNumberQuorumNodes` is set to a non-zero value. Default value is 0. See the section called “Node quorum management” for more details on the use of this configuration value.
`nodeQuorumVoteCount`	Integer	Number of votes for this node when calculating the node quorum percentage. This value is ignored if `nodeQuorum` is set to `Disable` or `nodeQuorumPercentage` is 0. Default value is 1. See the section called “Node quorum management” for more details on the use of this configuration value.
`keepAliveSendIntervalSeconds`	Integer	Keep-alive send interval. Must be a positive number. Default value is 1 second.
`nonResponseTimeoutSeconds`	Integer	Keep-alive non-response timeout interval following a send failure on a network interface to a remote node. When the non-response timeout expires on all configured interfaces, the remote node is marked down. Must be a positive number. Default value is 2 seconds.
`deferredWritesEnabled`	Boolean	Control deferred writes distribution protocol. A value of `true` causes writes to be deferred until a transaction commits. A value of `false` causes writes to be done immediately following a field modification. Default value is `true`.

Table 6.3, “Distributed transaction configuration” defines the distributed transaction configuration block values.

Table 6.3. Distributed transaction configuration

Name	Type	Description
`timeoutSeconds`	Integer	The number of seconds to wait for a distributed lock. If this time value expires, a deadlock is thrown and the transaction is retried. Must be a positive number. Default value is 60 seconds.
`numberCompletedTransactions`	Integer	The maximum number of committed or aborted transactions to retain for each remote node. This is used for recovery processing to determine the outcome of global transactions if a remote node crashes and is restarted with a loss of shared memory. Default value is 1000.

Table 6.4, “Distribution transport configuration” defines the distribution transport configuration block values. Distribution can be configured to use either TCP, SSL, or Infiniband as the underlying transport protocol. Listener addresses are specified using the listenerAddressList configuration value. The possible transport protocol specifiers are:

IPv4[/[TCP|SSL]] - Specify an IPv4 network address using either TCP or SSL as the underlying transport protocol. The TCP specifier is optional - both IPv4 and IPv4/TCP specify TCP. IPv4/SSL specifies SSL.
IPv6[/[TCP|SSL]] - Specify an IPv6 network address using either TCP or SSL as the underlying transport protocol. The TCP specifier is optional - both IPv6 and IPv6/TCP specify TCP. IPv6/SSL specifies SSL.
IPoSDP - specify an Infiniband transport address.

Table 6.4. Distribution transport configuration

Name	Type	Description
`listenerAddressList`	String array	Listener address list. A list of TCP or Infiniband (Linux only) network addresses on which to start a listener. Remote nodes communicate to this node using these listeners. The format of each specified network listener address is: `IPv4[/[TCP\|SSL]]\|IPv6[/[TCP\|SSL]]\|IPoSDP:``<host address>``:<port number>` The `<host address>` default value is all interfaces. The default `<port number>` is 5557. For example to listen on all `IPv4` interfaces on port number 5557 would require the following entry: `{IPv4::};` To listen on all `IPv6` interfaces on port 5558 using SSL would require the following entry: `{IPv6/SSL::5558};` To listen on a specific Infiniband interface would required the following entry: `{IPoSDP:myhostname:5558};` To listen on all of these interfaces requires the following entry: `{IPv4::,IPv6/SSL::5558,IPoSDP:myhostname:5558};`
`numberSearchPorts`	Integer	The number of ports to search before reporting a listener start failure. The search is started at the configured network listener port number and then incremented by one on each failure up to this value. A value of 0 disables port search. Default value is 20.
`nodeActiveTimeoutSeconds`	Integer	The amount of time in seconds to wait for a remote node to move into the `Active` state before a resource unavailable exception is raised. The wait is done for a partition's active node when defining a partition, or for each remote node when a `discover cluster` administration command is executed. This value must be > 0. Default value is 60 seconds.
`tcpNoDelayEnabled`	Boolean	Control enabling of `TCP_NODELAY` socket option on connection establishment. Default value is `true`.
`maximumPDUSizeBytes`	Integer	Control maximum protocol data unit (PDU) size used for distributed communications. This value must be > 4000. Default value is 1000000 bytes.

Table 6.5, “Dynamic discovery configuration” defines the dynamic discovery configuration block values. Dynamic discovery of nodes depends on service discovery being enabled. If service discovery is disabled, dynamic discovery cannot be enabled. See the section called “Installation” for details.

Table 6.5. Dynamic discovery configuration

Name	Type	Description
`enabled`	Boolean	Enable or disable dynamic discovery. Dyanmic discovery can only be enabled if the discovery service is enabled. Default value is true.

Table 6.6, “Static discovery configuration” defines the static discovery RemoteNode configuration block values.

The listener address of the remote node is specified using the networkAddressList configuration value. The transport protocol specifier in the networkAddressList uses the same format as the listenerAddressList described in Table 6.4, “Distribution transport configuration”.

Table 6.6. Static discovery configuration

Name	Type	Description
`name`	String	Remote node name. This field is required. There cannot be multiple configurations with the same node name.
`networkAddressList`	String array	Network address list. This field is required. A list of TCP or Infiniband (Linux only) network addresses over which the node can be reached. Connections to this remote node are done using round-robin in the order in which the addresses are specified in this list. The format of each specified network address is `IPv4[/[TCP\|SSL]]\|IPv6[/[TCP\|SSL]]\|IPoSDP:``<host address>:<port number>`. For example to connect over `IPv4` using TCP to an `acme.com` host at port 5557 would require the following entry: `{IPv4:acme.com:5557}`

Updating configuration

Distribution configuration can be updated on a running node to change any of the configuration values. The majority of the distribution configuration values can be updated without impacting distribution connectivity between nodes - the distribution services are not disabled. An active distribution configuration can be replaced with a new version with different configuration values. When replacing an active distribution configuration with an updated version, the configuration name must be the same. For example:

//
//  Original distribution configuration version
//
configuration "connectivity" version "1.0" type "distribution"
{
};

//
//  Updated distribution configuration version - has same name - connectivity
//
configuration "connectivity" version "2.0" type "distribution"
{
};

Changing these configuration values require the distribution services to be disabled first, by deactivating the current configuration:

The transport listenerAddressList (Table 6.4, “Distribution transport configuration”).
All dynamic discovery configuration values - enabled, broadcastPort, and broadcastHost (Table 6.5, “Dynamic discovery configuration”).
Removing an active statically defined remote node (Table 6.6, “Static discovery configuration”).

If an active distribution configuration is replaced with a new version with any of the above values changed, the activation of the new version will fail audit and the previous version will remain active.

Configuration Examples

Example 6.2, “High availability configuration” shows an example of a high availability configuration.

Example 6.2. High availability configuration

configuration "distribution" version "1.0" type "distribution"
{
    configure switchadmin
    {
        configure Distribution
        {
            HighAvailability
            {
                nodeQuorum = Enable;
                minimumNumberQuorumNodes = 1;
                keepAliveSendIntervalSeconds = 1;
                nonResponseTimeoutSeconds = 2;
            };
        };
    };
};

Example 6.3, “Distributed transaction configuration” shows an example of a distributed transaction configuration.

Example 6.3. Distributed transaction configuration

configuration "distribution" version "1.0" type "distribution"
{
     configure switchadmin
     {
         configure Distribution
         {
            Transaction
            {
                timeoutSeconds = 60;
                numberCompletedTransactions = 1000;
            };
         };
     };
};

Example 6.4, “Distribution transport configuration” shows an example of a distribution transport configuration.

Example 6.4. Distribution transport configuration

configuration "distribution" version "1.0" type "distribution"
{
    configure switchadmin
    {
        configure Distribution
        {
            Transport
            {
               listenerAddressList =
               {
                   "IPv4::7000", // listen on all IPv4 interfaces
                   "IPv6/SSL::7100:, // listen on all IPv6 interfaces and use SSL 
                   "IPoSDP:ib_nodeA:2000" // listen on Infiniband ib_nodeA interface
               };
               numberSearchPorts = 20;
               nodeActiveTimeoutSeconds = 60;
               tcpNoDelayEnabled = true;
               maximumPDUSizeBytes = 1000000;
            };         
        };
    };
};

Table 6.5, “Dynamic discovery configuration” shows an example of a dynamic discovery configuration.

Example 6.5. Dynamic discovery configuration

configuration "distribution" version "1.0" type "distribution"
{
    configure switchadmin
    {
        configure Distribution
        {
           DynamicDiscovery
           {
               enabled = true;
           };
        };
    };
};

Table 6.6, “Static discovery configuration” shows an example of a static discovery configuration.

Example 6.6. Static discovery configuration

configuration "distribution" version "1.0" type "distribution"
{
    configure switchadmin
    {
        configure Distribution
        {
            configure StaticDiscovery
            {
                RemoteNode
                {
                    name = "A";
                    networkAddressList =
                    {
                        "IPv4/SSL:a-networkaddress:22222", // IPv4 address using SSL
                        "IPv6:a-networkaddress:33333" // IPv6 address
                        
                    };
                };
                RemoteNode
                {
                    name = "B";
                    networkAddressList =
                    {
                        "IPoSDP:a-networkaddress:2000" // Infiniband address
                    };
                };
            };
        };
    };
};

This configuration defines two remote nodes named A and B. These remote nodes must have their distribution listener addresses configured to match the network addresses specified in the RemoteNode configuration block. Specifically node A must have specified the listenerAddressList in Example 6.7, “Node A transport configuration” and node B must have specified the listenerAddressList in Example 6.8, “Node B transport configuration”.


	Do not mix Infiniband and Ethernet addresses in the same `RemoteNode` configuration block. Network addresses in the `RemoteNode` `networkAddressList` configuration are used round-robin in the order they are specified. Mixing Infiniband and Ethernet addresses will use a mix of these two connection types to the remote node, defeating the performance advantages of using Infiniband.

Example 6.7. Node A transport configuration

configuration "node-a" version "1.0" type "distribution"
{
    configure switchadmin
    {
        configure Distribution
        {
            Transport
            {
               listenerAddressList =
               {
                   "IPv4/SSL:a-networkaddress:22222",
                   "IPv6:a-networkaddress:33333"
                   
               };
               ...
            };         
        };
    };
};

Example 6.8. Node B transport configuration

configuration "node-b" version "1.0" type "distribution"
{
    configure switchadmin
    {
        configure Distribution
        {
            Transport
            {
               listenerAddressList =
               {
                   "IPoSDP:a-networkaddress:2000"
               };
               ...
            };         
        };
    };
};