Runtime Node Configuration

< Previous		Next >

Overview

This article provides a reference for writing a StreamBase Runtime node deployment configuration file where the HOCON type is com.tibco.ep.streambase.configuration.node.

The node deploy configuration file is used to define deployment-time values, and to also override, and possibly augment, default application configuration. Deployment time configuration can be specified for multiple nodes in a single node deploy configuration file. The nodes can be in the same or different clusters. The node configuration to use from the node deploy configuration file when installing a node is determined by matching the node name being installed with a node name in the node deploy configuration file.

Required Header Lines

Each configuration file must contain the following header lines, typically found at the beginning of each file:

name

Specifies an arbitrary, case-sensitive string to name this configuration, which must be unique among other files with the same type, if any. Configuration files can refer to each other by this name. Select a name that reminds you of this configuration's type and purpose. For example:

name = "NodeDeployment"

version

Specifies an arbitrary version number that you can use to keep track of file versions for this configuration type in your development project. The maintenance of version numbers is under user control; StreamBase does not compare versions when loading configuration files during the fragment launch process. The version number is a string value, and can contain any combination of characters and numbers. For example:

version = "1.0.0"

type

This essential setting specifies the unique HOCON configuration type described on this page.

type = "com.tibco.ep.dtm.configuration.node"

The three header lines taken together constitute a unique signature for each HOCON file in a project's configurations folder. Each project's configurations folder can contain only one file with the same signature.

The top-level configuration object defines the configuration envelope the same way for all HOCON file types.

configuration

On a line below the header lines, enter the word configuration followed by an open brace. The configuration object is a sibling of the name, version, and type identifiers, and serves to define the configuration envelope around this type's objects as described on this page. The file must end with the matching close brace.

configuration = {
...
...
}

HOCON Properties Explained

Below shows the configuration's HOCON properties, usage, and syntax example, where applicable.

NodeDeploy

Node deployment configuration

globalConfiguration

String. An array of global late-bound application configuration objects. The configurations are loaded onto all nodes in the cluster. Format is identical to the configuration service HOCON configuration format, but in string form.

This array is optional and has no default value; it is used only during the boot process.

nodes

Associative node instances, keyed by node name.

A1.nodedeploy

String. Example of a node name containing the following objects:

description

Human-readable description.

For example:

"First node in the nodedeploy cluster for testing."

nodeType

The node's type, as defined in the application definition. This is a reference to a type defined in the application definition configuration file. If no type is found, the configuration will fail. This object is optional and its value references a built-in node type, default.

For example:

"nodetype1"

engines

Associative objects describing engines that run the fragments used by this node's nodeType. Each engine name is mapped to a descriptor that contains the ID of the fragment the engine is running, plus any engine-specific configurations. The identifier must be present in a single fragment archive's manifest TIBCO-EP-Fragment-Identifier property.

This object is optional and is used only during the boot process. If absent, the node will run a single engine for each fragment supported by its node type.

engine1

String. An example of an engine name, containing the following properties and arrays:

fragmentIdentifier

String. The binding's fragment identifier.

For example:

fragmentIdentifier = "fragment1"

configuration

String. An array of engine-specific initial late-bound application configuration objects. Configurations that inherit from EngineAffinityConfiguration are given a default engine affinity value of this engine. Format is identical to the configuration service HOCON configuration format, but in string form. This array is optional and has no default value.

For example:

configuration = []

communication

Communication ports. Optional. If unset, a set of default property values apply, which are set in the contain types.

numberSearchPorts

Long. The number of ports to search before reporting a distribution listener start failure. The search is started at the configured network listener port number for each distribution listener interface, and is then incremented by one on each failure up to this value. A value of 0 disables port search. The minimum value that can be specified is the total number of fragment engines plus one. Optional. Default value is 20.

This value in effect specifies a range size for a set of listener ports for auto-generated distribution listeners. If you specify a dataTransportPort value of n, the range of potentially used ports is n + numberSearchPorts value. Do not assign another dataTransportPort value for one node within the port range of another node. That is, you must stagger dataTransportPort assignments at least numberSearchPorts apart.

For example:

numberSearchPorts = 10

discoveryPort

Long. Broadcast discovery port. This property is optional and its value defaults to 54321.

For example:

discoveryPort = 2222

discoveryRequestAddresses

String. An array of broadcast discovery client request addresses, either IPV4 or DNS host name. The discovery requests are broadcast to the discovery port on the network interface associated with each network address. The discovery request recipients always listen on all interfaces; that is not configurable. This array is optional and its default value is a single address, which is the system host name.

For example:

discoveryRequestAddresses = [ "localhost" ].

administration

Communication settings for administration transport. This is optional. If unset, a set of default values apply. These defaults are set in the contain types.

address

String. Administration interface address. You can specify this optional property as an IPv4 or DNS address; this is used only during the boot process. The default of "" means listen on all addresses

For example:

address = "localhost"

transportPort

Long. Administration transport port number. This property is used only during the boot process, is optional, and its value defaults to 0, indicating that the node should auto-generate the port number.

For example:

transportPort = 1000

webEnable

Bool. Enable administration web server. This property is optional; the default is true indicating that the web interface is enabled.

For example:

webEnable = true

webPort

Long; {0–65535}. Administration web server listener port. This property is optional with a default of 0, indicating that indicating that the node will auto-generate the port number.

For example:

webPort = 0

webNodeFilters

String. Node filters for the web administration. Array of node filters specifying the nodes that can be managed by this node's web-based administration GUI. Each filter is a fully or partially qualified scoped service name. Optional; defaults to all nodes in the cluster.

For example:

webNodeFilters = [ "A1.nodedeploy" ].

distributionListenerInterfaces

Distribution transport. If this optional array is unset, a set of default values apply. These defaults are set in the contain types.

address

String. Listen interface address. This address can be specified as an IPv4, IPv6, or DNS name. A special prefix of IPoSDP: indicates the use of Infiniband sockets direct protocol (Linux only).

If this optional property is unset, the default is "" indicating listening on all interfaces.

For example, address = "localhost".

dataTransportPort

Long. Distribution listener port number. This property is optional and its value defaults to 0, indicating that the node should auto-generate the port number. Do not assign another dataTransportPort value for one node within the port range of another node, as described for the numberSearchPorts property above.

For example:

dataTransportPort = 1001

secure

Bool. A secure-transport indicator. If true, use TLS to secure communication to the host, if false do not. This property is optional and its default value is false.

For example:

secure = false

proxyDiscovery

In the case where node discovery cannot be achieved at runtime (for example, UDP broadcast is not permitted), you can configure node discovery here as a proxy discovery.

remoteNodes

String. List of remote nodes to which to provide proxy discovery services.

For example:

remoteNodes = [ "A2.nodedeploy" ]

configuration

An array of node-specific initial late-bound application configuration objects.

availabilityZoneMemberships

An array of availability zones that this node is part of.

staticzone

The memberships are an associative array keyed by availability zone. In this example there is a downstream availability zone called staticzone. This node is declaring membership in this zone and binding that membership to a set of partitions.

staticPartitionBindings

Static partition binding object.

For example:

staticPartitionBindings = {
  P1 = {
   type = ACTIVE
   restoreFrom = true
   replication = SYNCHRONOUS
   }
}

P1

String. Example partition name. For a partition named P1 containing the following properties:

type

Valid types are ACTIVE and REPLICA. property is required.

restoreFrom

Specify that the partition should be restored from this node. When this property is set to true, the partition defined on this node is loaded to the local node. This should be done when restoring a node from a split-brain situation, where this node is the node in the cluster where all objects should be preserved, and the local node is the node being restored. Any conflicts during restore will preserve the objects on this node, and remove the conflicting objects on the local node.

A restore is needed when multiple nodes are currently the active node for a partition in a cluster due to a split-brain scenario. In this case, the application needs to decide which active node will be the node where the objects are preserved during a restore. Note that this node does not necessarily have to be the node that becomes the partition's active node after the restore completes.

The actual restore of the partition is done in the enable() or enablePartitions() method when the JOIN_CLUSTER_RESTORE enableAction is used. If any other enableAction is used, object data is not preserved, and no restoration of partition objects is done.

If restoreFrom is not set after a split-brain scenario, the runtime performs a cluster-wide broadcast to find the current active node, and use that node to restore instances in the partition. If multiple active nodes are found, the first responder is chosen.

This optional property's default value is false.

replication

Replication type. Valid values are SYNCHRONOUS and ASYNCHRONOUS. This property is optional and its default value is SYNCHRONOUS.

dynamiczone

The memberships are an association keyed by availability zone. In this case there is a downstream availability zone called dynamiczone. This node is declaring membership in this zone and binding that membership to a set of partitions.

votes

Long. Number of quorum votes for this node. Optional. If not set, a default of 1 is used.

For example, votes = 1

dynamicPartitionBinding

Dynamic partition binding.

type

Member type. For dynamic groups, valid types are PRIMARY and BACKUP.

For example, type = PRIMARY

availabilityZones

An associative array of availability zones, keyed by zone name. Each node can be part of zero or more zones.

staticzone

The memberships are an associative array keyed by availability zone. In this case there is a downstream availability zone called staticzone. This node is declaring membership in this zone and binding that membership to a set of partitions.

dataDistributionPolicy

Data distribution policy that applies to this availability zone. This is a reference to a policy defined in the application definition file. If no policy is found, the configuration will fail.

For example:

dataDistributionPolicy = "static"

staticPartitionPolicy

Static partition distribution policy. This optional object has no default value.

disableOrder

Order when disabling partitions in static partition groups.

Value is one of REVERSE_CONFIGURATION or REVERSE_LOCAL_THEN_REMOTE.

REVERSE_CONFIGURATION disables the partitions in reverse order based on partition rank.

REVERSE_LOCAL_THEN_REMOTE disables all partitions that have the local node as the active node, then disable all partitions where a remote node is the active node.

This property is optional and its default value is REVERSE_CONFIGURATION.

enableOrder

Order when enabling partitions in static partition groups.

Value is one of CONFIGURATION or REMOTE_THEN_LOCAL.

CONFIGURATION enables the partitions based on partition rank.

REMOTE_THEN_LOCAL enables all partitions that have a remote node as the active node, then enables the partitions where the active node is the local node.

This property is optional and its default value is CONFIGURATION.

loadOnNodesPattern

String. A regular expression describing which nodes know about the partitions in this group. Note this set of nodes could be different from the nodes that actually participate in the partitions. Load-on interest can be expressed in one or both of two ways: via this regular expression or explicitly by each node.

If the local node matches loadOnNodesPattern, then the partition is added to the local node even if it is not the active or backup node. This allows creation of foreign partitions. Note that the configuration is additive; you can define a partition in the node's availability zone membership, the availability zone loadOnNodesPattern, or both.

This property is optional and its value defaults to all nodes that are members of this availability zone.

For example:

loadOnNodesPattern = ".*"

staticPartitions

An associative array of partitions, keyed by partition name.

P1

String. Example partition name. For a partition named P1 containing the property:

rank

Long. The partition's ranking number for enable and disable order.

When the partition staticPartitionEnableOrder or staticPartitionDisableOrder is set to CONFIGURATION, this rank specifies the order. Partitions with a lower ranking number are enabled before those with a higher ranking number.

If multiple partitions share the same ranking number, their enable order is indeterminate.

The highAvailability, staticPartitionEnableOrder, and staticPartitionDisableOrder properties control whether ranking order is strictly observed, or whether local partitions are always enabled ahead of remote with ranking observed within those classifications.

Disable order always reverses the rankings, with higher numbers disabled ahead of lower numbers.

This property is optional and its default value is 1.

For example:

rank = 1

dynamiczone

The memberships are an associative array keyed by availability zone. In this case there is a downstream availability zone called one called dynamiczone. This node is declaring membership in that zone and binding that membership to a set of partitions.

percentageOfVotes

Long. Minimum percentage of votes needed to form a quorum. This object is optional and has no default value. This object is mutually exclusive with minimumNumberOfVotes. If neither is set then quorums are not enabled for this availability zone.

For example:

percentageOfVotes = 51

dataDistributionPolicy

Data distribution policy that applies to this availability zone. This is a reference to a policy defined in the application definition file. If no policy is found, the configuration will fail.

For example:

dataDistributionPolicy = "dynamic"

dynamicPartitionPolicy

Dynamic partition distribution policy. This object is optional and has no default value.

primaryMemberPattern

String. A Java regular expression describing the nodes that comprise this group's primary membership. This property is optional and defaults to the ".*" value.

For example:

primaryMemberPattern = ".*"

backupMemberPattern

String. A regular expression describing the group's backup membership. This property is optional and has no default value.

For example:

backupMemberPattern = ".*"

minimumNumberOfVotes

Long. Minimum number of votes needed to form a quorum. This property is optional, has no default value, and is mutually exclusive with percentageOfVotes. If neither are set then quorums are not enabled for this availability zone.

For example:

minimumNumberOfVotes = 5

quorumMemberPattern

String. Quorum membership pattern. This is a Java regular expression Membership can be expressed in one or both of two ways: via this regular expression or explicitly by each node.

All nodes matching this regular expression are part of the quorum. Such members get a single quorum vote. This property has no default value.

For example:

quorumMemberPattern = ".*"

Configuration File Sample

The following is an example of the com.tibco.ep.dtm.configuration.node file type.

name = "NodeDeployment"
version = "1.0"
type = "com.tibco.ep.dtm.configuration.node"
configuration = {
  NodeDeploy = {
    globalConfiguration = [
      ]
    nodes = {
      "A1.nodedeploy" = {
         description = "my node"
         nodeType = "nodetype1"
         engines = {
           engine1 = {
             fragmentIdentifier = "fragment1"
             configuration = []
           }
         }
         communication = {
           numberSearchPorts = 10
           discoveryPort = 2222
           discoveryRequestAddresses = [ 
             ${HOSTNAME} 
           ]
           administration = {
             address = ${HOSTNAME}
             transportPort = ${A1_ADMINPORT}
             webEnable = true
             webPort = 0
             webNodeFilters = [ 
               "A1.nodedeploy" 
             ]
           }
           distributionListenerInterfaces = [ 
             {
                address = ${HOSTNAME}
                dataTransportPort = ${A1_DATATRANSPORTPORT}
                secure = false
             } 
           ]
           proxyDiscovery = {
             remoteNodes = [ 
               "A1.nodedeploy" 
             ]
           }
         }
           configuration = [
           ]
           availabilityZoneMemberships = {
             staticzone = {
               staticPartitionBindings = {
                 P1 = {
                   type = ACTIVE
                   restoreFrom = true
                   replication = SYNCHRONOUS
                 }
              }
           }
             dynamiczone = {
               votes = 1
               dynamicPartitionBinding = {
                 type = PRIMARY
               }
             }
           }
            }
      "A2.nodedeploy" = {
         nodeType = "nodetype1"
         communication = {
           distributionListenerInterfaces = [ {
             address = ${HOSTNAME}
             dataTransportPort = ${A2_DATATRANSPORTPORT}
             secure = false
           } ]
         }
      }
      "B1.nodedeploy" = {
         nodeType = "nodetype1"
         communication = {
           distributionListenerInterfaces = [ {
             address = ${HOSTNAME}
             dataTransportPort = ${B1_DATATRANSPORTPORT}
             secure = false
           } ]
         }
      }

    }
        
    availabilityZones = {
      staticzone = {
        dataDistributionPolicy = "static"
        staticPartitionPolicy = {
          disableOrder = REVERSE_CONFIGURATION
          enableOrder = CONFIGURATION
          staticPartitions = {
            P1 = {
              rank = 1
            }
          }
          loadOnNodesPattern = ".*"
        }
    }
      dynamiczone = {
        percentageOfVotes = 51
        dataDistributionPolicy = "dynamic"
        dynamicPartitionPolicy = {
          primaryMemberPattern = ".*"
          backupMemberPattern = ".*"
        }
        minimumNumberOfVotes = 5
        quorumMemberPattern = ".*"
      }
    }
  }
}

Node Availability Zone Configuration Examples

If a node is explicitly bound to a dynamic availability zone, using the availabilityZoneMemberships object, that zone is used. However, if a node is not explicitly bound to a zone then an implicit binding can occur if primaryMemberPattern or backupMemberPattern match the node name.

The following examples describe explicit and implicit zone bindings.

Explicit binding example:

name = "HAConfig"
version = "1.0"
type = "com.tibco.ep.dtm.configuration.node"
configuration = {
  NodeDeploy = {
    nodes = {
      "A1.nodedeploy" = {
          availabilityZoneMemberships = {
            dynamiczone = {                  // use this zone
              dynamicPartitionBinding = {
                type = PRIMARY
              }
            }
          }
      }
    }
    availabilityZones = {
        dynamiczone = {
            dataDistributionPolicy = "dynamic"
        }
    }
  }
}

Implicit binding example:

version = "1.0"
type = "com.tibco.ep.dtm.configuration.node"
configuration = {
  NodeDeploy = {
    nodes = {
      "A1.nodedeploy" = {
      }
    }
    availabilityZones = {
      dynamiczone = {
        dataDistributionPolicy = "dynamic"
          dynamicPartitionPolicy = {
            primaryMemberPattern = "A[14].nodedeploy"  
            // nodes matching this pattern are included in the zone
          }
      }
    }
  }
}

Since the primaryMemberPattern property defaults to .*, the implicit example can be simplified to:

name = "HAConfig"
version = "1.0"
type = "com.tibco.ep.dtm.configuration.node"
configuration = {
  NodeDeploy = {
    nodes = {
      "A1.nodedeploy" = {
      }
    }
    availabilityZones = {
      dynamiczone = {
        dataDistributionPolicy = "dynamic"
        // matches dynamicPartitionBinding .* by default
        }
    }
  }
}