Runtime Node Configuration

< Previous		Next >

Overview

This article provides a reference for writing a StreamBase Runtime Node configuration file.

The node deploy configuration file is used to define deployment-time values, and to also override, and possibly augment, default application configuration. Deployment time configuration can be specified for multiple nodes in a single node deploy configuration file. The nodes can be in the same or different clusters. The node configuration to use from the node deploy configuration file when installing a node is determined by matching the node name being installed with a node name in the node deploy configuration file.

Required Header Lines

Each configuration file must contain the following header lines, typically found at the beginning of each file:

name

Specifies an arbitrary, case-sensitive string to name this configuration, which must be unique among other files with the same type, if any. Configuration files can refer to each other by this name. Select a name that reminds you of this configuration's type and purpose. For example:

name = "NodeDeployment"

version

Specifies an arbitrary version number that you can use to keep track of file versions for this configuration type in your development project. The maintenance of version numbers is under user control; StreamBase does not compare versions when loading configuration files during the fragment launch process. The version number is a string value, and can contain any combination of characters and numbers. For example:

version = "1.0.0"

type

This essential setting specifies the unique HOCON configuration type described on this page.

type = "com.tibco.ep.dtm.configuration.node"

The three header lines taken together constitute a unique signature for each HOCON file in a project's configurations folder. Each project's configurations folder can contain only one file with the same signature.

The top-level configuration object defines the configuration envelope the same way for all HOCON file types.

configuration

On a line below the header element lines, enter the word configuration followed by an open brace. The configuration element is a sibling of the name, version, and type elements, and serves to define the configuration envelope around this type's objects as described on this page. The file must end with the matching close brace.

configuration = {
...
...
}

HOCON Elements Explained

Below shows the configuration's HOCON elements, its name-values, usage, and syntax example, where applicable.

NodeDeploy

Node deployment configuration

globalConfiguration

String. An array of global late-bound application configuration objects. The configurations are loaded onto all nodes in the cluster. Format is identical to the configuration service HOCON configuration format, but in string form.

This name-value pair is optional and has no default value; it is used only during the boot process.

nodes

An associative array of node instances, keyed by node name.

"A1.nodedeploy"

String. Example of a node, which contains the following name-value pairs:

description

Human-readable description.

For example:

"my node"

nodeType

The node's type, as defined in the application definition. This is a reference to a type defined in the application definition file. If no type is found, the configuration will fail. This name-value pair is optional and its value references a built-in node type "default".

For example:

"nodetype1"

engines

An associative array describing engines that run the fragments used by this node's node type. Each engine name is mapped to a descriptor that contains the ID of the fragment the engine is running, plus any engine-specific configurations. The identifier must be present in a single fragment archive's manifest "TIBCO-EP-Fragment-Identifier" property.

This name-value pair is optional. If it is missing, the node will run a single engine for each fragment supported by its node type. This name-value pair is used only during the boot process.

engine1

String. An example of an engine name, containing the following name-value pairs:

fragmentIdentifier

String. The binding's fragment identifier.

For example:

fragmentIdentifier = "fragment1"

configuration

String. An array of engine-specific initial late-bound application configuration objects. Configurations that inherit from EngineAffinityConfiguration are given a default engine affinity value of this engine. Format is identical to the configuration service HOCON configuration format, but in string form. This name-value pair is optional and has no default value.

For example:

configuration = []

communication

Communication ports. This object is optional. If unset, a set of default property values apply. These defaults are set in the container types.

numberSearchPorts

Long. The number of ports to search before reporting a distribution listener start failure. The search is started at the configured network listener port number for each distribution listener interface, and is then incremented by one on each failure up to this value. A value of 0 disables port search. The minimum value that can be specified is the total number of fragment engines plus one. Optional. Default value is 20.

This value in effect specifies a range size for a set of listener ports for auto-generated distribution listeners. If you specify a dataTransportPort value of n, the range of potentially used ports is n + numberSearchPorts. Do not assign another dataTransportPort value for one node within the port range of another node. That is, you must stagger dataTransportPort assignments at least numberSearchPorts apart.

For example:

numberSearchPorts = 10

discoveryPort

Long. Broadcast discovery port. This name-value pair is optional and its value defaults to 54321.

For example:

discoveryPort = 2222

discoveryRequestAddresses

String. An array of broadcast discovery client request addresses, either IPV4 or DNS host name. The discovery requests are broadcast to the discovery port on the network interface associated with each network address. The discovery request recipients always listen on all interfaces; that is not configurable. This name-value pair is optional and its default value is a single address which is the system host name.

For example:

discoveryRequestAddresses = [ "localhost" ].

administration

Communication settings for administration transport. This is optional. If unset, a set of default values apply. These defaults are set in the contain types.

address

String. Administration interface address. This name-value pair can be specified as an IPv4 or DNS address. This name-value pair is used only during the boot process. This name-value pair is optional, the default of "" is to listen on all addresses

For example:

address = "localhost"

transportPort

Long. Administration transport port number. This name-value pair is used only during the boot process. This name-value pair is optional and its value defaults to 0, indicating that the node should auto-generate the port number

For example:

transportPort = 1000

webEnable

Bool. Enable administration web server. This name-value pair is optional; the default is true indicating that the web interface is enabled.

For example:

webEnable = true

webPort

Long. Administration web server listener port. This name-value pair is optional; the default is 0 indicating that the port number is auto-assigned.

For example:

webPort = 0

webNodeFilters

String. Node filters for the web administration. List of node filters specifying the nodes that can be managed by this node's web-based administration GUI. Each filter is a fully or partially qualified scoped service name. This name-value pair is optional, and defaults to all nodes in the cluster.

For example:

webNodeFilters = [ "A1.nodedeploy" ].

distributionListenerInterfaces

Distribution transport. This is optional. If unset, a set of default values apply. These defaults are set in the contain types.

address

String. Listen interface address. This address can be specified as an IPv4, IPv6, or DNS name. A special prefix of "IPoSDP:" indicates the use of Infiniband sockets direct protocol (Linux only).

This name-value pair is optional. If unset the default is "" indicating listening on all interfaces.

For example, address = "localhost".

dataTransportPort

Long. Distribution listener port number. This property is optional and its value defaults to 0, indicating that the node should auto-generate the port number. Do not assign another dataTransportPort value for one node within the port range of another node, as described for the numberSearchPorts property above.

For example:

dataTransportPort = 1001

secure

Bool. A secure-transport indicator. If true, use TLS to secure communication to the host, if false do not. This name-value pair is optional and its default value is false.

For example:

secure = false

proxyDiscovery

In the case where node discovery can't be achieved at runtime (for example, UDP broadcast is not permitted), node discovery can be configured here as a proxy discovery.

remoteNodes

String. List of remote nodes to which to provide proxy discovery services.

For example:

remoteNodes = [ "A2.nodedeploy" ]

configuration

An array of node-specific initial late-bound application configuration objects.

availabilityZoneMemberships

Availability zones that this node is part of.

staticzone

The memberships are an associative array keyed by availability zone. In this case there is a downstream availability zone called "staticzone". This node is declaring membership in this zone and binding that membership to a set of partitions.

staticPartitionBindings

Static partition binding.

For example:

staticPartitionBindings = {
  P1 = {
   type = ACTIVE
   restoreFrom = true
   replication = SYNCHRONOUS
   }
}

P1

String. Example partition name. For a partition named P1 containing the following name-value pairs:

type

Valid types are ACTIVE and REPLICA. This name-value pair is required.

restoreFrom

Specify that the partition should be restored from this node. When this property is set to true, the partition defined on this node is loaded to the local node. This should be done when restoring a node from a split-brain situation, where this node is the node in the cluster where all objects should be preserved, and the local node is the node being restored. Any conflicts during restore will preserve the objects on this node, and remove the conflicting objects on the local node.

A restore is needed when multiple nodes are currently the active node for a partition in a cluster due to a split-brain scenario. In this case, the application needs to decide which active node will be the node where the objects are preserved during a restore. Note that this node does not necessarily have to be the node which becomes the partition's active node after the restore completes.

The actual restore of the partition is done in the enable() or enablePartitions() method when the JOIN_CLUSTER_RESTORE EnableAction is used. If any other EnableAction is used, object data isn't preserved, and no restoration of partition objects is done.

If restoreFromNode isn't set after a split-brain scenario, the runtime will perform a cluster wide broadcast to find the current active node, and use that node to restore instances in the partition. If multiple active nodes are found, the first responder is chosen.

This name-value pair is optional and has default value of false.

replication

Replication type. Valid values are SYNCHRONOUS and ASYNCHRONOUS. This name-value pair is optional and its default value is SYNCHRONOUS.

dynamiczone

The memberships are an associative array keyed by availability zone. In this case there is a downstream availability zone called "dynamiczone". This node is declaring membership in this zone and binding that membership to a set of partitions.

votes

Long. Number of quorum votes for this node. Optional. If not set, a default of 1 is used.

For example, votes = 1

dynamicPartitionBinding

Dynamic partition binding.

type

Member type. For dynamic groups, valid types are PRIMARY and BACKUP.

For example, type = PRIMARY

availabilityZones

An associative array of availability zones, keyed by zone name. Each node can be part of zero or more zones.

staticzone

The memberships are an associative array keyed by availability zone. In this case there is a downstream availability zone called "staticzone". This node is declaring membership in this zone and binding that membership to a set of partitions.

dataDistributionPolicy

Data distribution policy that applies to this availability zone. This is a reference to a policy defined in the application definition file. If no policy is found, the configuration will fail.

For example:

dataDistributionPolicy = "static"

staticPartitionPolicy

Static partition distribution policy. This name-value pair is optional and has no default value.

disableOrder

Order when disabling partitions in static partition groups.

Value is one of REVERSE_CONFIGURATION or REVERSE_LOCAL_THEN_REMOTE.

REVERSE_CONFIGURATION disables the partitions in reverse order based on partition rank.

REVERSE_LOCAL_THEN_REMOTE disables all partitions that have the local node as the active node, then disable all partitions where a remote node is the active node.

This name-value pair is optional and its default value is REVERSE_CONFIGURATION.

enableOrder

Order when enabling partitions in static partition groups.

Value is one of CONFIGURATION or REMOTE_THEN_LOCAL.

CONFIGURATION enables the partitions based on partition rank.

REMOTE_THEN_LOCAL enables all partitions that have a remote node as the active node, then enables the partitions where the active node is the local node.

This name-value pair is optional and its default value is CONFIGURATION.

loadOnNodesPattern

String. A regular expression describing which nodes know about the partitions in this group. Note this set of nodes could be different from the nodes that actually participate in the partitions. Load-on interest can be expressed in one or both of two ways: via this regular expression or explicitly by each node.

If the local node matches loadOnNodesPattern, then the partition is added to the local node even if it is not the active or backup node. This allows creation of sparse partitions. Note that the configuration is additive; a partition can be defined in the node's availability zone membership, the availability zone loadOnNodesPattern, or both.

This name-value pair is optional and its value defaults to all nodes that are members of this availability zone.

For example:

loadOnNodesPattern = ".*"

staticPartitions

An associative array of partitions, keyed by partition name.

P1

String. Example partition name. For a partition named P1 containing the following name-value pair:

rank

Long. The partition's ranking number for enable and disable order.

When the partition staticPartitionEnableOrder or staticPartitionDisableOrder is set to CONFIGURATION, this rank specifies the order. Partitions with a lower ranking number are enabled before those with a higher ranking number.

If multiple partitions share the same ranking number, their enable order is indeterminate.

The highAvailability staticPartitionEnableOrder and staticPartitionDisableOrder properties control whether ranking order is strictly observed, or whether local partitions are always enabled ahead of remote with ranking observed within those classifications.

Disable order always reverses the rankings, with higher numbers disabled ahead of lower numbers.

This name-value pair is optional and its default value is 1.

For example:

rank = 1

dynamiczone

The memberships are an associative array keyed by availability zone. In this case there is a downstream availability zone called one called "dynamiczone". This node is declaring membership in that zone and binding that membership to a set of partitions.

percentageOfVotes

Long. Minimum percentage of votes needed to form a quorum. This name-value pair is optional and has no default value. This name-value pair is mutually exclusive with minimumNumberOfVotes. If neither is set then quorums are not enabled for this availability zone.

For example:

percentageOfVotes = 51

dataDistributionPolicy

Data distribution policy that applies to this availability zone. This is a reference to a policy defined in the application definition file. If no policy is found, the configuration will fail.

For example:

dataDistributionPolicy = "dynamic"

dynamicPartitionPolicy

Dynamic partition distribution policy. This name-value pair is optional and has no default value.

primaryMemberPattern

String. A Java regular expression describing the nodes that comprise this group's primary membership. This name-value pair is optional and defaults to .*.

For example:

primaryMemberPattern = ".*"

backupMemberPattern

String. A regular expression describing the group's backup membership. This name-value pair is optional and has no default value.

For example:

backupMemberPattern = ".*"

minimumNumberOfVotes

Long. Minimum number of votes needed to form a quorum. This name-value pair is optional and has no default value. This name-value pair is mutually exclusive with percentageOfVotes. If neither are set then quorums are not enabled for this availability zone.

For example:

minimumNumberOfVotes = 5

quorumMemberPattern

String. Quorum membership pattern. This is a Java regular expression Membership can be expressed in one or both of two ways: via this regular expression or explicitly by each node.

All nodes matching this regular expression are part of the quorum. Such members get a single quorum vote. This name-value pair has no default value.

For example:

quorumMemberPattern = ".*"

Configuration File Sample

The following is an example of the com.tibco.ep.dtm.configuration.node file type.

name = "NodeDeployment"
version = "1.0"
type = "com.tibco.ep.dtm.configuration.node"
configuration = {
    NodeDeploy = {
    
        globalConfiguration = [
        ]
                
        nodes = {
        
            "A1.nodedeploy" = {
                description = "my node"
                nodeType = "nodetype1"
                engines = {
                    engine1 = {
                      fragmentIdentifier = "fragment1"
                      configuration = []
                    }
                }
                communication = {
                    numberSearchPorts = 10
                    discoveryPort = 2222
                    discoveryRequestAddresses = [ 
                        ${HOSTNAME} 
                    ]
                    administration = {
                        address = ${HOSTNAME}
                        transportPort = ${A1_ADMINPORT}
                        webEnable = true
                        webPort = 0
                        webNodeFilters = [ 
                            "A1.nodedeploy" 
                        ]
                    }
                    distributionListenerInterfaces = [ 
                        {
                            address = ${HOSTNAME}
                            dataTransportPort = ${A1_DATATRANSPORTPORT}
                            secure = false
                        } 
                    ]
                    proxyDiscovery = {
                        remoteNodes = [ 
                            "A1.nodedeploy" 
                        ]
                    }
                }
                
                configuration = [
                ]
                
                availabilityZoneMemberships = {
                    staticzone = {
                        staticPartitionBindings = {
                            P1 = {
                                type = ACTIVE
                                restoreFrom = true
                                replication = SYNCHRONOUS
                            }
                        }
                    }
                    dynamiczone = {
                        votes = 1
                        dynamicPartitionBinding = {
                            type = PRIMARY
                        }
                    }
                }
            }
            "A2.nodedeploy" = {
                nodeType = "nodetype1"
                communication = {
                    distributionListenerInterfaces = [ {
                        address = ${HOSTNAME}
                        dataTransportPort = ${A2_DATATRANSPORTPORT}
                        secure = false
                    } ]
                }
            }
            "B1.nodedeploy" = {
                nodeType = "nodetype1"
                communication = {
                    distributionListenerInterfaces = [ {
                        address = ${HOSTNAME}
                        dataTransportPort = ${B1_DATATRANSPORTPORT}
                        secure = false
                    } ]
                }
            }

        }
        
        availabilityZones = {
            staticzone = {
                dataDistributionPolicy = "static"
                staticPartitionPolicy = {
                    disableOrder = REVERSE_CONFIGURATION
                    enableOrder = CONFIGURATION
                    staticPartitions = {
                        P1 = {
                            rank = 1
                        }
                    }
                    loadOnNodesPattern = ".*"
                }
             }
            dynamiczone = {
                percentageOfVotes = 51
                dataDistributionPolicy = "dynamic"
                dynamicPartitionPolicy = {
                    primaryMemberPattern = ".*"
                    backupMemberPattern = ".*"
                }
                minimumNumberOfVotes = 5
                quorumMemberPattern = ".*"
             }
        }

    }
}

Node Availability Zone Configuration Examples

If a node is explicitly bound to a dynamic availability zone, using the availabilityZoneMemberships name-value pair, that zone is used. However, if a node is not explicitly bound to a zone then an implicit binding can occur if primaryMemberPattern or backupMemberPattern match the node name.

The following examples describe explicit and implicit zone bindings.

Explicit binding example:

name = "HAConfig"
version = "1.0"
type = "com.tibco.ep.dtm.configuration.node"
configuration = {
    NodeDeploy = {
        nodes = {
            "A1.nodedeploy" = {
                availabilityZoneMemberships = {
                    dynamiczone = {                  // use this zone
                        dynamicPartitionBinding = {
                            type = PRIMARY
                        }
                    }
                }
            }
        }
        availabilityZones = {
            dynamiczone = {
                dataDistributionPolicy = "dynamic"
            }
        }
    }
}

Implicit binding example:

version = "1.0"
type = "com.tibco.ep.dtm.configuration.node"
configuration = {
    NodeDeploy = {
        nodes = {
            "A1.nodedeploy" = {
            }
        }
        availabilityZones = {
            dynamiczone = {
                dataDistributionPolicy = "dynamic"
                dynamicPartitionPolicy = {
                    primaryMemberPattern = "A[14].nodedeploy"  
                    // nodes matching this pattern are included in the zone
                }
            }
        }
    }
}

Since the primaryMemberPattern name-value pair defaults to .*, the implicit example can be simplified to:

name = "HAConfig"
version = "1.0"
type = "com.tibco.ep.dtm.configuration.node"
configuration = {
    NodeDeploy = {
        nodes = {
            "A1.nodedeploy" = {
            }
        }
        availabilityZones = {
            dynamiczone = {
                dataDistributionPolicy = "dynamic"
                // matches dynamicPartitionBinding .* by default
            }
        }
    }
}