- Index

< Previous |  | Next >

# Runtime Node Configuration

## Overview

This article provides a reference for writing a StreamBase Runtime node deployment configuration file where the HOCON type is `com.tibco.ep.streambase.configuration.node`.

The node deploy configuration file is used to define deployment-time values, and to also override, and possibly augment, default application configuration. Deployment time configuration can be specified for multiple nodes in a single node deploy configuration file. The nodes can be in the same or different clusters. The node configuration to use from the node deploy configuration file when installing a node is determined by matching the node name being installed with a node name in the node deploy configuration file.

## Required Header Lines

Each configuration file must contain the following header lines, typically found at the beginning of each file:

name

  Specifies an arbitrary, case-sensitive string to name this configuration, which must be unique among other files with the same type, if any. Configuration files can refer to each other by this name. Select a name that reminds you of this configuration's type and purpose. For example:

      name = "NodeDeployment"

version

  Specifies an arbitrary version number that you can use to keep track of file versions for this configuration type in your development project. The maintenance of version numbers is under user control; StreamBase does not compare versions when loading configuration files during the fragment launch process. The version number is a string value, and can contain any combination of characters and numbers. For example:

      version = "1.0.0"

type

  This essential setting specifies the unique HOCON configuration type described on this page.

      type = "com.tibco.ep.dtm.configuration.node"

The three header lines taken together constitute a unique signature for each HOCON file in a project's `configurations` folder. Each project's `configurations` folder can contain only one file with the same signature.

The top-level `configuration` object defines the configuration envelope the same way for all HOCON file types.

configuration

  On a line below the header lines, enter the word `configuration` followed by an open brace. The `configuration` object is a sibling of the `name`, `version`, and `type` identifiers, and serves to define the configuration envelope around this type's objects as described on this page. The file must end with the matching close brace.

      configuration = {
      ...
      ...
      }

## HOCON Properties Explained

Below shows the configuration's HOCON properties, usage, and syntax example, where applicable.

NodeDeploy

  Node deployment configuration

  globalConfiguration

    String. An array of global late-bound application configuration objects. The configurations are loaded onto all nodes in the cluster. Format is identical to the configuration service HOCON configuration format, but in string form.

    This array is optional and has no default value; it is used only during the boot process.

  nodes

    Associative node instances, keyed by node name.

    `A1.nodedeploy`

      String. Example of a node name containing the following objects:

      description

        Human-readable description.

        For example:

            "First node in the nodedeploy cluster for testing."

      nodeType

        The node's type, as defined in the application definition. This is a reference to a type defined in the application definition configuration file. If no type is found, the configuration will fail. This object is optional and its value references a built-in node type, `default`.

        For example:

            "nodetype1"

      engines

        Associative objects describing engines that run the fragments used by this node's `nodeType`. Each engine name is mapped to a descriptor that contains the ID of the fragment the engine is running, plus any engine-specific configurations. The identifier must be present in a single fragment archive's manifest `TIBCO-EP-Fragment-Identifier` property.

        This object is optional and is used only during the boot process. If absent, the node will run a single engine for each fragment supported by its node type.

        `engine1`

          String. An example of an engine name, containing the following properties and arrays:

          fragmentIdentifier

            String. The binding's fragment identifier.

            For example:

                fragmentIdentifier = "fragment1"

          configuration

            String. An array of engine-specific initial late-bound application configuration objects. Configurations that inherit from EngineAffinityConfiguration are given a default engine affinity value of this engine. Format is identical to the configuration service HOCON configuration format, but in string form. This array is optional and has no default value.

            For example:

                configuration = []

      communication

        Communication ports. Optional. If unset, a set of default property values apply, which are set in the contain types.

        numberSearchPorts

          Long. The number of ports to search before reporting a distribution listener start failure. The search is started at the configured network listener port number for each distribution listener interface, and is then incremented by one on each failure up to this value. A value of 0 disables port search. The minimum value that can be specified is the total number of fragment engines plus one. Optional. Default value is 20.

          This value in effect specifies a range size for a set of listener ports for auto-generated distribution listeners. If you specify a `dataTransportPort` value of n, the range of potentially used ports is n + `numberSearchPorts` value. Do not assign another `dataTransportPort` value for one node within the port range of another node. That is, you must stagger `dataTransportPort` assignments at least `numberSearchPorts` apart.

          For example:

              numberSearchPorts = 10

        discoveryPort

          Long. Broadcast discovery port. This property is optional and its value defaults to 54321.

          For example:

              discoveryPort = 2222

        discoveryRequestAddresses

          String. An array of broadcast discovery client request addresses, either IPV4 or DNS host name. The discovery requests are broadcast to the discovery port on the network interface associated with each network address. The discovery request recipients always listen on all interfaces; that is not configurable. This array is optional and its default value is a single address, which is the system host name.

          For example:

              discoveryRequestAddresses = [ "localhost" ].

        administration

          Communication settings for administration transport. This is optional. If unset, a set of default values apply. These defaults are set in the contain types.

          address

            String. Administration interface address. You can specify this optional property as an IPv4 or DNS address; this is used only during the boot process. The default of `""` means listen on all addresses

            For example:

                address = "localhost"

          transportPort

            Long. Administration transport port number. This property is used only during the boot process, is optional, and its value defaults to 0, indicating that the node should auto-generate the port number.

            For example:

                transportPort = 1000

          webEnable

            Bool. Enable administration web server. This property is optional; the default is true indicating that the web interface is enabled.

            For example:

                webEnable = true

          webPort

            Long; {0–65535}. Administration web server listener port. This property is optional with a default of 0, indicating that indicating that the node will auto-generate the port number.

            For example:

                webPort = 0

          webNodeFilters

            String. Node filters for the web administration. Array of node filters specifying the nodes that can be managed by this node's web-based administration GUI. Each filter is a fully or partially qualified scoped service name. Optional; defaults to all nodes in the cluster.

            For example:

                webNodeFilters = [ "A1.nodedeploy" ].

        distributionListenerInterfaces

          Distribution transport. If this optional array is unset, a set of default values apply. These defaults are set in the contain types.

          address

            String. Listen interface address. This address can be specified as an IPv4, IPv6, or DNS name. A special prefix of `IPoSDP:` indicates the use of Infiniband sockets direct protocol (Linux only).

            If this optional property is unset, the default is "" indicating listening on all interfaces.

                For example, address = "localhost".

          dataTransportPort

            Long. Distribution listener port number. This property is optional and its value defaults to 0, indicating that the node should auto-generate the port number. Do not assign another `dataTransportPort` value for one node within the port range of another node, as described for the `numberSearchPorts` property above.

            For example:

                dataTransportPort = 1001

          secure

            Bool. A secure-transport indicator. If true, use TLS to secure communication to the host, if false do not. This property is optional and its default value is false.

            For example:

                secure = false

        proxyDiscovery

          In the case where node discovery cannot be achieved at runtime (for example, UDP broadcast is not permitted), you can configure node discovery here as a proxy discovery.

          remoteNodes

            String. List of remote nodes to which to provide proxy discovery services.

            For example:

                remoteNodes = [ "A2.nodedeploy" ]

      configuration

        An array of node-specific initial late-bound application configuration objects.

      availabilityZoneMemberships

        An array of availability zones that this node is part of.

        staticzone

          The memberships are an associative array keyed by availability zone. In this example there is a downstream availability zone called `staticzone`. This node is declaring membership in this zone and binding that membership to a set of partitions.

          staticPartitionBindings

            Static partition binding object.

            For example:

                staticPartitionBindings = {
                  P1 = {
                   type = ACTIVE
                   restoreFrom = true
                   replication = SYNCHRONOUS
                   }
                }

            P1

              String. Example partition name. For a partition named `P1` containing the following properties:

              type

                Valid types are ACTIVE and REPLICA. property is required.

              restoreFrom

                Specify that the partition should be restored from this node. When this property is set to true, the partition defined on this node is loaded to the local node. This should be done when restoring a node from a split-brain situation, where this node is the node in the cluster where all objects should be preserved, and the local node is the node being restored. Any conflicts during restore will preserve the objects on this node, and remove the conflicting objects on the local node.

                A restore is needed when multiple nodes are currently the active node for a partition in a cluster due to a split-brain scenario. In this case, the application needs to decide which active node will be the node where the objects are preserved during a restore. Note that this node does not necessarily have to be the node that becomes the partition's active node after the restore completes.

                The actual restore of the partition is done in the enable() or enablePartitions() method when the JOIN_CLUSTER_RESTORE `enableAction` is used. If any other enableAction is used, object data is not preserved, and no restoration of partition objects is done.

                If `restoreFrom` is not set after a split-brain scenario, the runtime performs a cluster-wide broadcast to find the current active node, and use that node to restore instances in the partition. If multiple active nodes are found, the first responder is chosen.

                This optional property's default value is false.

              replication

                Replication type. Valid values are SYNCHRONOUS and ASYNCHRONOUS. This property is optional and its default value is SYNCHRONOUS.

        dynamiczone

          The memberships are an association keyed by availability zone. In this case there is a downstream availability zone called `dynamiczone`. This node is declaring membership in this zone and binding that membership to a set of partitions.

          votes

            Long. Number of quorum votes for this node. Optional. If not set, a default of 1 is used.

                For example, votes = 1

          dynamicPartitionBinding

            Dynamic partition binding.

            type

              Member type. For dynamic groups, valid types are PRIMARY and BACKUP.

                  For example, type = PRIMARY

  availabilityZones

    An associative array of availability zones, keyed by zone name. Each node can be part of zero or more zones.

    staticzone

      The memberships are an associative array keyed by availability zone. In this case there is a downstream availability zone called `staticzone`. This node is declaring membership in this zone and binding that membership to a set of partitions.

      dataDistributionPolicy

        Data distribution policy that applies to this availability zone. This is a reference to a policy defined in the application definition file. If no policy is found, the configuration will fail.

        For example:

            dataDistributionPolicy = "static"

      staticPartitionPolicy

        Static partition distribution policy. This optional object has no default value.

        disableOrder

          Order when disabling partitions in static partition groups.

          Value is one of REVERSE_CONFIGURATION or REVERSE_LOCAL_THEN_REMOTE.

          REVERSE_CONFIGURATION disables the partitions in reverse order based on partition rank.

          REVERSE_LOCAL_THEN_REMOTE disables all partitions that have the local node as the active node, then disable all partitions where a remote node is the active node.

          This property is optional and its default value is REVERSE_CONFIGURATION.

        enableOrder

          Order when enabling partitions in static partition groups.

          Value is one of CONFIGURATION or REMOTE_THEN_LOCAL.

          CONFIGURATION enables the partitions based on partition rank.

          REMOTE_THEN_LOCAL enables all partitions that have a remote node as the active node, then enables the partitions where the active node is the local node.

          This property is optional and its default value is CONFIGURATION.

        loadOnNodesPattern

          String. A regular expression describing which nodes know about the partitions in this group. Note this set of nodes could be different from the nodes that actually participate in the partitions. Load-on interest can be expressed in one or both of two ways: via this regular expression or explicitly by each node.

          If the local node matches `loadOnNodesPattern`, then the partition is added to the local node even if it is not the active or backup node. This allows creation of foreign partitions. Note that the configuration is additive; you can define a partition in the node's availability zone membership, the availability zone `loadOnNodesPattern`, or both.

          This property is optional and its value defaults to all nodes that are members of this availability zone.

          For example:

              loadOnNodesPattern = ".*"

        staticPartitions

          An associative array of partitions, keyed by partition name.

          P1

            String. Example partition name. For a partition named `P1` containing the property:

            rank

              Long. The partition's ranking number for enable and disable order.

              When the partition `staticPartitionEnableOrder` or `staticPartitionDisableOrder` is set to CONFIGURATION, this rank specifies the order. Partitions with a lower ranking number are enabled before those with a higher ranking number.

              If multiple partitions share the same ranking number, their enable order is indeterminate.

              The `highAvailability`, `staticPartitionEnableOrder`, and `staticPartitionDisableOrder` properties control whether ranking order is strictly observed, or whether local partitions are always enabled ahead of remote with ranking observed within those classifications.

              Disable order always reverses the rankings, with higher numbers disabled ahead of lower numbers.

              This property is optional and its default value is 1.

              For example:

                  rank = 1

    dynamiczone

      The memberships are an associative array keyed by availability zone. In this case there is a downstream availability zone called one called `dynamiczone`. This node is declaring membership in that zone and binding that membership to a set of partitions.

      percentageOfVotes

        Long. Minimum percentage of votes needed to form a quorum. This object is optional and has no default value. This object is mutually exclusive with `minimumNumberOfVotes`. If neither is set then quorums are not enabled for this availability zone.

        For example:

            percentageOfVotes = 51

      dataDistributionPolicy

        Data distribution policy that applies to this availability zone. This is a reference to a policy defined in the application definition file. If no policy is found, the configuration will fail.

        For example:

            dataDistributionPolicy = "dynamic"

      dynamicPartitionPolicy

        Dynamic partition distribution policy. This object is optional and has no default value.

        primaryMemberPattern

          String. A Java regular expression describing the nodes that comprise this group's primary membership. This property is optional and defaults to the "`.*`" value.

          For example:

              primaryMemberPattern = ".*"

        backupMemberPattern

          String. A regular expression describing the group's backup membership. This property is optional and has no default value.

          For example:

              backupMemberPattern = ".*"

      minimumNumberOfVotes

        Long. Minimum number of votes needed to form a quorum. This property is optional, has no default value, and is mutually exclusive with `percentageOfVotes`. If neither are set then quorums are not enabled for this availability zone.

        For example:

            minimumNumberOfVotes = 5

      quorumMemberPattern

        String. Quorum membership pattern. This is a Java regular expression Membership can be expressed in one or both of two ways: via this regular expression or explicitly by each node.

        All nodes matching this regular expression are part of the quorum. Such members get a single quorum vote. This property has no default value.

        For example:

            quorumMemberPattern = ".*"

## Configuration File Sample

The following is an example of the com.tibco.ep.dtm.configuration.node file type.

    name = "NodeDeployment"
    version = "1.0"
    type = "com.tibco.ep.dtm.configuration.node"
    configuration = {
      NodeDeploy = {
        globalConfiguration = [
          ]
        nodes = {
          "A1.nodedeploy" = {
             description = "my node"
             nodeType = "nodetype1"
             engines = {
               engine1 = {
                 fragmentIdentifier = "fragment1"
                 configuration = []
               }
             }
             communication = {
               numberSearchPorts = 10
               discoveryPort = 2222
               discoveryRequestAddresses = [
                 ${HOSTNAME}
               ]
               administration = {
                 address = ${HOSTNAME}
                 transportPort = ${A1_ADMINPORT}
                 webEnable = true
                 webPort = 0
                 webNodeFilters = [
                   "A1.nodedeploy"
                 ]
               }
               distributionListenerInterfaces = [
                 {
                    address = ${HOSTNAME}
                    dataTransportPort = ${A1_DATATRANSPORTPORT}
                    secure = false
                 }
               ]
               proxyDiscovery = {
                 remoteNodes = [
                   "A1.nodedeploy"
                 ]
               }
             }
               configuration = [
               ]
               availabilityZoneMemberships = {
                 staticzone = {
                   staticPartitionBindings = {
                     P1 = {
                       type = ACTIVE
                       restoreFrom = true
                       replication = SYNCHRONOUS
                     }
                  }
               }
                 dynamiczone = {
                   votes = 1
                   dynamicPartitionBinding = {
                     type = PRIMARY
                   }
                 }
               }
                }
          "A2.nodedeploy" = {
             nodeType = "nodetype1"
             communication = {
               distributionListenerInterfaces = [ {
                 address = ${HOSTNAME}
                 dataTransportPort = ${A2_DATATRANSPORTPORT}
                 secure = false
               } ]
             }
          }
          "B1.nodedeploy" = {
             nodeType = "nodetype1"
             communication = {
               distributionListenerInterfaces = [ {
                 address = ${HOSTNAME}
                 dataTransportPort = ${B1_DATATRANSPORTPORT}
                 secure = false
               } ]
             }
          }

        }

        availabilityZones = {
          staticzone = {
            dataDistributionPolicy = "static"
            staticPartitionPolicy = {
              disableOrder = REVERSE_CONFIGURATION
              enableOrder = CONFIGURATION
              staticPartitions = {
                P1 = {
                  rank = 1
                }
              }
              loadOnNodesPattern = ".*"
            }
        }
          dynamiczone = {
            percentageOfVotes = 51
            dataDistributionPolicy = "dynamic"
            dynamicPartitionPolicy = {
              primaryMemberPattern = ".*"
              backupMemberPattern = ".*"
            }
            minimumNumberOfVotes = 5
            quorumMemberPattern = ".*"
          }
        }
      }
    }

### Node Availability Zone Configuration Examples

If a node is explicitly bound to a dynamic availability zone, using the `availabilityZoneMemberships` object, that zone is used. However, if a node is not explicitly bound to a zone then an implicit binding can occur if `primaryMemberPattern` or `backupMemberPattern` match the node name.

The following examples describe explicit and implicit zone bindings.

Explicit binding example:

    name = "HAConfig"
    version = "1.0"
    type = "com.tibco.ep.dtm.configuration.node"
    configuration = {
      NodeDeploy = {
        nodes = {
          "A1.nodedeploy" = {
              availabilityZoneMemberships = {
                dynamiczone = {                  // use this zone
                  dynamicPartitionBinding = {
                    type = PRIMARY
                  }
                }
              }
          }
        }
        availabilityZones = {
            dynamiczone = {
                dataDistributionPolicy = "dynamic"
            }
        }
      }
    }

Implicit binding example:

    version = "1.0"
    type = "com.tibco.ep.dtm.configuration.node"
    configuration = {
      NodeDeploy = {
        nodes = {
          "A1.nodedeploy" = {
          }
        }
        availabilityZones = {
          dynamiczone = {
            dataDistributionPolicy = "dynamic"
              dynamicPartitionPolicy = {
                primaryMemberPattern = "A[14].nodedeploy"
                // nodes matching this pattern are included in the zone
              }
          }
        }
      }
    }

Since the `primaryMemberPattern` property defaults to `.*`, the implicit example can be simplified to:

    name = "HAConfig"
    version = "1.0"
    type = "com.tibco.ep.dtm.configuration.node"
    configuration = {
      NodeDeploy = {
        nodes = {
          "A1.nodedeploy" = {
          }
        }
        availabilityZones = {
          dynamiczone = {
            dataDistributionPolicy = "dynamic"
            // matches dynamicPartitionBinding .* by default
            }
        }
      }
    }