Design and Implementation

This section describes the design of the Broker service and its implementation.

Instance Status Descriptions

During the instance registration and deregistration processes, instances undergo the following statuses:

  • Pending: Indicates that an instance is registered but verification is pending to confirm whether the instance has started successfully without errors.

  • Active: Signifies that an instance is registered and currently active.

  • Inactive: Indicates that an instance is not currently active. An instance is marked as inactive by the ActiveInstanceMonitor flag when it has not communicated or pinged back for an extended period.

Database Details

The following table shows the database details for Broker service.

Instance Ledger
Name Description Data Type
Oracle DB PostgreSQL DB
instance_id Unique identifier for the registered instance VARCHAR2(255) character varying(255)
ip_address IP address associated with a specific instance VARCHAR2(250) character varying(250)
port Port number associated with a particular service or endpoint, specifies the network port used for communication. VARCHAR2(20) character varying(20)
status Current operational status or state of a system component or service, aiding in monitoring and management by indicating whether it is active, inactive NUMBER(10, 0) numeric(10, 0)
service_name The name or identifier of a specific service, aiding in the categorization and organization of services within the system. VARCHAR2(50) character varying(50)
last_updated_timestamp Last timestamp for instance status updated NUMBER(20, 0) numeric(20, 0)

The database scripts are present at the following locations:

PostgreSQL database: <OM_HOME>/db/dbscripts/postgreSQL/broker

Oracle database: <OM_HOME>/db/dbscripts/oracle/broker

Configuration Value Details

Properties File Update

Before the server startup, the Broker service uses the configurator component to retrieve essential configuration properties from the application.properties file, which ensures a seamless startup process.

 
Property Name Property Value Description
server.port 9105 Port at which the broker service starts.
configuratorServiceUrl http://localhost:9090 Configurator base URL to connect and download configuration properties and configuration files.
configuratorServiceRetryCount 5 Number of retries to perform in case of request failure to configurator.
configuratorServiceRetryDuration 5 Delay between each retry, in seconds.
configuratorTrustStoreAbsoluteFilePath C:/Users/cacert Trust store file path if configurator is on HTTPS.
configuratorTrustStorePassword tibco123 Trust store password if configurator is on HTTPS.
configuratorTrustStoreType jks Trust store type if configurator is on HTTPS.
security.key ENC(nSa0k6lmjPPN8ZA5SO6BpQ==) Security key to securely connect to Configurator. The default value is encrypted of 1t1s@asy.

Update Broker Service configuration values

The configurator must have certain configuration properties available. Before the server starts, you must upload these configuration properties to the configurator. These properties are available in the $OM_HOME/seed-data/app-properties/ConfigValues_BrokerService.json file.

Catalog Client Configuration

When the Broker service triggers pending southbound notifications, it populates the plan item details in some of those notifications. To achieve this, the Broker service connects to the Catalog Service and fetches the latest plan fragment details associated with the plan item, then includes these details in the southbound notifications.

Conversely, the Orchestrator fetches these notifications by directly connecting to the catalog database and caching the details in its memory. Unlike the Orchestrator, the Broker service retrieves data through REST calls to the Catalog service and does not cache the responses.

 
Property Name Property Value Description
catalogServiceBaseUrl http://localhost:9092 Catalog Service base URL.
catalogTrustStorePassword tibco123 Trust store password if Catalog is on HTTPS.
catalogTrustStoreType jks Trust Store type if Catalog is on HTTPS.
catalogTrustStoreFileName cacert Trust Store certificate file name. This is required if the catalog is running on HTTPS. The Broker Service downloads this file from the configurator.

Broker Database Configuration

The Broker Service manages instance life-cycles by storing instance information in a database. For more information, see the Instance Management APIs.

 
Property Name Property Value Description
brokerDsUrl jdbc:postgresql://localhost:5432/brokerdbll?
currentSchema=brokerschemall
JDBC URL of the broker database.
brokerDsUsername brokeruserll User name of the broker database.
brokerDsPassword brokeruserll Password of the broker database.
brokerDsInitializeSize 2 Number of connections that are to be established when the connection pool is started.
brokerDsMaxIdle 11 Maximum number of connections that must be kept in the idle pool.
brokerDsMaxActive 12 Maximum number of active connections that can be allocated from this pool at the same time.
brokerDsMaxWait 1000 Maximum number of milliseconds that the pool waits when there are no available connections.
brokerDsTestOnBorrow false Enable connection validation before being borrowed from the pool.
brokerDsValidationInterval 5000 Data source validation interval in milliseconds.
brokerDsTestWhileIdle true Enable connection validation while idle in connection pool.
brokerDsTimeBetweenEvictionRunsMillis 5000 Minimum amount of time in milliseconds an object must sit idle in the pool before it is eligible for eviction.
brokerDsMinEvictableIdleTimeMillis 5000 Minimum amount of time in milliseconds an object might sit idle in the pool.
brokerDsNumTestsPerEvictionRun 5 Maximum number of connections to examine during each evictor run.
brokerDsDefaultAutoCommit false Enable auto commit after each transaction.
brokerDsRollbackOnReturn false Enable transaction rollback when connection is returned back to the pool.
brokerDsCommitOnReturn false Enable transaction commit when connection is returned back to the pool.
brokerDsCustomProperty   Pooled Database Custom Property.

Instance Management Configuration

The Broker service includes job schedulers that monitor instance life-cycles. The following properties are required for these jobs. For more information, see Instance Status Management and Instance Failure Mechanism.

 
Property Name Property Value Description
activeInstanceMonitorInterval 20 Time interval, measured in minutes, at which ActiveInstanceMonitor operates or runs.
activeInstanceMonitorThreshold 5 The duration parameter used by ActiveInstanceMonitor to identify and declare inactive instances based on the timestamp comparison (LastUpdatedTimestamp) against the defined threshold.
inactiveInstanceMonitorCronExpression 0 */20 * * * ? A Cron expression that determines the execution schedule of InactiveInstanceMonitor.
orderEventTransferTaskQueue com.tibco.broker.order.event.task.queue Queue on OrderEventTransfer task would be published internally by Broker Service.
orderEventTransferTaskListenerCount 10 Number of concurrent listener on the orderEventTransferTaskQueue.
orderEventTransferTaskRetryCount 5 Retry count for orderEventTransferTask.
orderEventTransferTaskRetryInterval 1 Retry interval between orderEventTransferTask .
timeDependencyMonitorInterval 1 Fixed interval in minutes to monitor time dependency with null instance id.

No Originator Listener Configuration

The Broker maintains a connection to the EMS server to listen for southbound replies. This includes listening for replies from currently inactive instances and those lacking the required originator header. Additionally, to ensure the Broker service can recover requests sent to the southbound system, it must be configured with the appropriate queues.

For more information, see the following topics:

 
Property Name Property Value Description
orchestratorInboundNoOriginatorQueue tibco.aff.orchestrator.inbound.no.originator.queue Queue on which Orchestrator listens for incoming replies from the southbound system.
orchestratorInboundNoOriginatorDeadQueue tibco.aff.orchestrator.inbound.no.originator.dead.queue Dead Queue on which Orchestrator sends the unprocessable messages.
orchestratorInboundNoOriginatorReceiverCount 5 Number of concurrent consumers on each southbound response queue.
orchestratorInboundNoOriginatorRetryCount 5 Number of retries in case of any exception.
orchestratorInboundNoOriginatorRetryInterval 5000 Delay in Milliseconds between consecutive retries.

Recovery Notification Configuration

Broker service requires the following properties to process recovery notifications. For more information, see Recovery Notification Processing.

 
Property Name Property Value Description
recoveryNotificationQueue tibco.aff.orchestrator.order.event.notification.queue The queue to listen to the recovery notification sent by Orchestrator.
recoveryNotificationDeadQueue tibco.aff.orchestrator.order.event.notification.queue.dead Dead Queue to send the unprocessable recovery notification.
recoveryNotificationReceiverCount 5 Number of listeners to listen to the recovery notification.
recoveryNotificationRetryCount 5 Number of retries in case of failure.
recoveryNotificationRetryInterval 5000 Interval in milliseconds between each retry.

Update Orchestrator Service Configuration Values

You need to update the Orchestrator service configuration values. The properties are available in the $OM_HOME/seed-data/app-properties/ConfigValues_OrchService.json file.

Order Database Configuration

Broker service interacts with the order database to carry out the following functions:

Property Name Property Value Description
orderDsUrl

jdbc:postgresql://localhost:5432/

orderdbllhf7?currentSchema=orderschemallhf7

JDBC URL of the broker database.
orderDsUsername orderuserll User name of the broker database.
orderDsPassword orderuserll Password of the broker database.
orderDsInitializeSize 2 Number of connections that are to be established when the connection pool is started.
orderDsMaxIdle 11 Maximum number of connections that must be kept in the idle pool.
orderDsMaxActive 12 Maximum number of active connections that can be allocated from this pool at the same time.
orderDsMaxWait 1000 Maximum number of milliseconds that the pool waits when there are no available connections.
orderDsTestOnBorrow false Enable connection validation before being borrowed from the pool.
orderDsValidationInterval 5000 Data source validation interval in milliseconds.
orderDsTestWhileIdle true Enable connection validation while idle in connection pool.
orderDsTimeBetweenEvictionRunsMillis 5000 Minimum amount of time in milliseconds an object must sit idle in the pool before it is eligible for eviction.
orderDsMinEvictableIdleTimeMillis 5000 Minimum amount of time in milliseconds an object might sit idle in the pool.
orderDsNumTestsPerEvictionRun 5 Maximum number of connections to examine during each evictor run.
orderDsDefaultAutoCommit false Enable auto commit after each transaction.

Common Database Configuration

Property Name Property Value Description
datasourceDriverClassName org.postgresql.Driver Data Source Driver Class Name.
datasourceValidationQuery SELECT 1 SQL query that is used to validate connections.
hibernateDialect org.hibernate.dialect.PostgreSQLDialect Hibernate Dialect.
hibernateShowSql false Hibernate Show SQL.

EMS Configuration

The Broker stays connected to the EMS Server to listen for southbound replies, especially for instances that are inactive now and for replies lacking the required originator header.

 
Property Name Property Value Description
emsServerURL tcp://localhost:7222 EMS Server URL
emsServerUsername admin EMS Server User Name
emsServerPassword admin EMS Server Password
jndiConnectionFactory GenericConnectionFactory JNDI Connection Factory Name
tibjmsNamingSecurityProtocol   SSL configuration value, to be set to 'ssl'.
tibjmsNamingSslEnableVerifyHost   Set to 'false' to disable host verification in SSL.
initialContextFactory com.tibco.tibjms.naming.
TibjmsInitialContextFactory
Name of the Initial Context Factory.
jmsSessionTransacted true Determines if JMS sessions are transacted. Set to 'true'.
timeoutReceiveCalls 10000 Specifies the timeout for acknowledging EMS messages, in milliseconds.

Messaging Queues

The Broker stays connected to the EMS Server to listen for Southbound replies, especially for instances that are inactive now and for replies lacking the required originator header. Additionally, to recover requests sent to the Southbound system, the Broker requires proper queues configuration.

For more information, see the following topics:

 
Property Name Property Value Description
orchestratorInboundQueue tibco.aff.orchestrator.inbound.queue

The Broker service monitors the Orchestrator inbound queue as part of the Instance Failure Mechanism to manage messages in the event of an instance failure.

Exposed REST Endpoints

Instance Management APIs

The instance management APIs are as follows:

Registration of a new instance

URI: v1/instance/register

Request Body:

{
  "instanceId": "string",
  "serviceName": "string",
  "ipAddress": "string",
  "port": 0
}
 Request Schema
Parameter Description
serviceName Service name of the instance. Currently only Orchestrator is supported.
ipAddress IP address of the instance.
port Port of the instance.
instanceId Id of the instance.

Description

This API handles instance registration requests. After receiving a registration request, the Broker service uses a database sequence to create an instanceId. Subsequently, it performs validations before inserting a new record in the instance_ledger table.

Importantly, during registration, the Broker service does not immediately verify the instance's health status.

The initial status of the selected instance remains Pending. The instance status is updated to Active or Inactive based on the health assessment during the ping operation.

On successful registration, an Instance Status Change Notification is dispatched.

If a registration request is received for an already registered IP address and port, the Broker service returns the existing instance ID. This ensures that all pending messages on the EMS can be processed by the same instance that was restarted.

Request Validations and corresponding error responses

  • IP Address cannot be null or empty string.

    {
      "status": "BAD_REQUEST",
      "message": "Ip Address cannot be null or empty"
    }
    
  • Port cannot be less than or equal to zero

    {
      "status": "BAD_REQUEST",
      "message": "Port cannot be less than or equals to Zero"
    }
    
  • Service Name cannot be null or empty string

    {
      "status": "BAD_REQUEST",
      "message": "Service name cannot be null or empty"
    }
    
  • Service Name cannot be anything other than Orchestrator

    {
      "status": "BAD_REQUEST",
      "message": "Invalid service name {serviceName}"
    }
    

Response Body

{
  "instanceId": "string",
  "serviceName": "string",
  "ipAddress": "string",
  "port": int,
  "status": "string"
}
 Response Schema
Parameter Description
instanceId A unique identification of the instance about to be registered.
serviceName Service name of the instance. Currently only Orchestrator is supported.
ipAddress IP address of the instance.
port Port of the instance.
status
  • Pending

  • Active

  • Pending Purge

  • Inactive

 Responses
HTTP Status Description
201 New Instance is registered successfully.
200 Instance with ipAddress and port already existed with inactive status. Updated the status to Pending.
202 Instance with ipAddress and port already existed with active status. No operation is performed by Broker.
401 Invalid token
400
  • IpAddress cannot be null or empty

  • Port cannot be less than or equals to Zero

  • Service name cannot be null or empty

  • Invalid <<serviceName>>

    In place of <<serviceName>>, the actual name of the service is displayed.

Unregistration of an existing instance

URI: v1/instance/un-register/{instance_id}

Path Variable:

instance_id (Instance Id): A unique identification of the instance about to be unregistered. Accepted values can be any non null or non blank string.

Description

This asynchronous operation accepts the instance_id as a path variable and deregisters an instance from the Broker service. The deregistration process does not immediately remove the instance. The status of the instance is updated to Inactive.

Request Validations and corresponding error responses

  • Instance Id cannot be null or empty string

    {
      "status": "BAD_REQUEST",
      "message": "Instance Id cannot null or empty"
    }
    
  • Instance with id {{instance_id}} is not present

    {
      "status": "BAD_REQUEST",
      "message": "Instance with id {{instance_id}} is not present"
    }
    
 Responses
HTTP Status Description
202 Request accepted
401 Invalid token
400 Instance Id cannot null or empty
404 Instance with id {{instance_id}} is not present

Instance Ping Operation

URI: v1/instance/ping

Request Body

{
	"instanceId": "string",
	"status": "string",
	"inactiveComponentList": [
		"string"
	]
}
 Request Schema
Parameter Description
instanceId A unique identification of the instance
status

Health status of the instance. The following values are Allowed:

  • UP

  • DOWN

  • UNKNOWN

  • OUT_OF_SERVICE

inactiveComponentList List of components within the instance that do not currently exhibit an active health status.

Description

This API allows instances to regularly update their health status by communicating with the Broker Service. Instances compute their health status periodically and send this data to the Broker Service using the REST API.

After receiving these updates, the Broker Service processes and updates the instance's status in the database. If an instance reports an Up status, it is labeled as Active; otherwise, it is marked as Inactive.

In cases where an instance is Inactive, the system triggers an Order Event Transfer task to EMS for further handling.

Request Validations and corresponding error responses

  • Instance Id cannot be null or empty string

    {
      "status": "BAD_REQUEST",
      "message": "Instance Id cannot null or empty"
    }
    
  • Instance with id {{instance_id}} is not present

    {
      "status": "NOT_FOUND",
      "message": "Instance with id {{instance_id}} is not present"
    }
    
  • Status cannot be null or empty

    {
      "status": "BAD_REQUEST",
      "message": "Status cannot be null or empty"
    }
    
  • Invalid status

    {
      "status": "BAD_REQUEST",
      "message": "Invalid status. Only Up, Down, OUT_OF_SERVICE and UNKNOWN is allowed"
    }
    
 Responses
HTTP Status Description
200 Request processed
401 Invalid token
400
  • Instance Id cannot null or empty.

  • Status cannot be null or empty.

  • Invalid status. Only Up, Down, OUT_OF_SERVICE and UNKNOWN is allowed.

404 Instance with id {{instance_id}} is not present.

Instance clean up

URI: v1/instance/cleanup

Request Body: [ "string" ]

Request Schema: List of instance ids to purge from the database.

Description

In some scenarios, an instance might become inactive without being removed from the Broker database. You can use this API to manually specify a list of instance IDs that must be purged from the database.

If any instance in the provided list is not in an Inactive status, it is not purged. However, any other instances listed is removed from the database.

Request Validations and corresponding error responses

  • Instance Id list cannot be empty

    {
      "status": "BAD_REQUEST",
      "message": "Instance Id list cannot be empty"
    }
    
  • Are any of the given instances available with broker service?

    {
      "status": "NOT_FOUND",
      "message": "None of requested instances are present"
    }
    
  • Are all the given instances still active?

    {
      "status": "BAD_REQUEST",
      "message": "All of the requested instances are still active. Unregister them first."
    }
    
 Responses
HTTP Status Description
200 All requested instances are purged from the database.
206 Only some of the requested instances are purged from the database. This would usually occur when some instances are still active.
401 Invalid token
400
  • Instance Id list cannot be empty

  • All of the requested instances are still active.

  • Unregister them first

404 None of request instances are present

Get Instance Details By Instance Id

URI: v1/instance/{instanceId}

Path Variable:

instance_id (Instance Id): An exclusive identifier associated with the instance for which details are being sought. The accepted values can be any non null or non blank string.

Description

This API accepts an instanceId as a path variable and returns the details of the instance that is registered with the broker service.

Request Validations and corresponding error responses

  • Instance Id cannot be null or empty string

    {
      "status": "BAD_REQUEST",
      "message": "Instance Id cannot be null or empty string"
    }
    
  • Instance with id <<instanceId>> is not present

    {
      "status": "NOT_FOUND",
      "message": "Instance with id <<instanceId>> is not present"
    }
    

Response Body

{
  "instanceId": "abc",
  "serviceName": "Orchestrator",
  "ipAddress": "0.0.0.0",
  "port": 9093,
  "status": "UP"
}
 Response Schema
Parameter Description
instanceId A unique identification of the instance about to be registered.
serviceName Service name of the instance. Currently only Orchestrator is supported.
ipAddress IP address of the instance.
port Port of the instance.
status
  • Pending

  • Active

  • Pending Purge

  • Inactive

 Responses
HTTP Status Description
200 Returned the instance details of the requested instance.
401 Invalid token
400 Instance Id cannot be null or empty string.
404 Instance with id <<instanceId>> is not present.

Get Instance Details By Service Name

URI: /v1/instance?serviceName={serviceName}&pageNo={pageNo}&instancesPerPage={instancesPerPage}

Parameters

Description
serviceName (Service Name) The service name for which details of every available instance are being requested. The accepted values can be any non-null or non-blank string.
pageNo (Page Number) This parameter specifies the page number of the paginated data. The accepted values can be positive integers starting from 1, where 1 represents the first page.
instancesPerPage (Instances Per Page) This parameter determines the number of instances displayed on a single page of paginated data. The accepted values can be positive integers indicating the desired number of instances per page.

Description

You can use this API endpoint to retrieve a paginated list of records from the system for a given serviceName. You can specify the service name, page number, and the number of records to display per page using query parameters.

Request Validations and corresponding error responses

  • Service name cannot be null or empty string

    {
      "status": "BAD_REQUEST",
      "message": "Service name cannot be null or empty"
    }
    
  • No instance is available for the given service name

    {
      "status": "NOT_FOUND",
      "message": "No instances found for service name {serviceName}"
    }
    
  • Invalid Page number. Page number cannot be less than 1.

    {
      "status": "BAD_REQUEST",
      "message": "PageNo cannot be less than 1"
    }
    
  • Invalid instances per page. InstancesPerPage must be between 1 and 20.

    {
      "status": "BAD_REQUEST",
      "message": "InstancesPerPage should be between 1 and 20."
    }
    

Response Body

{
	"instanceDetailsList": [
		{
			"instanceId": "abc",
			"serviceName": "Orchestrator",
			"ipAddress": "0.0.0.0",
			"port": 9093,
			"status": "UP"
		}
	],
	"totalPages": 1,
	"currentPage": 1,
	"instancesPerPage": 10
}
 Response Schema
Parameter Description
instanceDetailsList List of the instance details
totalPages Total number of pages available as per instancesPerPage
currentPage Current Page number
instancesPerPage Instances per page requested by user
 Instance Details
Parameter Description
instanceId A unique identification of the instance about to be registered.
serviceName Service name of the instance. Currently only Orchestrator is supported.
ipAddress IP address of the instance
port Port of the instance
status
  • Pending

  • Active

  • Pending Purge

  • Inactive

 Responses
HTTP Status Description
200 Returns all instance details for all the instances registered with requested service name.
401 Invalid token
400
  • Service name cannot be null or empty

  • PageNo cannot be less than 1

  • InstancesPerPage must be between 1 and 20

404 No instances found for service name {serviceName}

Get Instance Details of all instances

URI: v1/instance/all?pageNo={pageNo}&instancesPerPage={instancesPerPage}

Parameters

Description
pageNo (Page Number) This parameter specifies the page number of the paginated data. The accepted values can be positive integers starting from 1, where 1 represents the first page.
instancesPerPage (Instances Per Page) This parameter determines the number of instances displayed on a single page of paginated data. The accepted values can be positive integers indicating the desired number of instances per page.

Description

You can use this API endpoint to retrieve a paginated list of records from the system. Also, you can specify the page number and the number of records to display per page using query parameters.

Request Validations and corresponding error responses

  • Invalid Page number. Page number cannot be less than 1

    {
      "status": "BAD_REQUEST",
      "message": "PageNo cannot be less than 1"
    }
    
  • Invalid instances per page. InstancesPerPage must be between 1 and 20.

    {
      "status": "BAD_REQUEST",
      "message": "InstancesPerPage should be between 1 and 20."
    }
    

Response Body

{
	"instanceDetailsList": [
		{
			"instanceId": "abc",
			"serviceName": "Orchestrator",
			"ipAddress": "0.0.0.0",
			"port": 9093,
			"status": "UP"
		}
	],
	"totalPages": 1,
	"currentPage": 1,
	"instancesPerPage": 10
}
 Response Schema
Parameter Description
instanceDetailsList List of the instance details
totalPages Total number of pages available as per instancesPerPage.
currentPage Current Page number
instancesPerPage Instances per page requested by user
 Instance Details
Parameter Description
instanceId A unique identification of the instance about to be registered.
serviceName Service name of the instance. Currently only Orchestrator is supported.
ipAddress IP address of the instance
port Port of the instance
status
  • Pending

  • Active

  • Pending Purge

  • Inactive

 Responses
HTTP Status Description
200 Returned all instance details for all the instances registered with requested service name.
401 Invalid token
400
  • PageNo cannot be less than 1

  • InstancesPerPage must be between 1 and 20

404 No instances available

Get all services

URI: /v1/instance/services

Description

This API provides a distinct list of service names for which instances are currently available within the Broker service.

Request Validations and corresponding error responses

  • No instance is available

    {
      "status": "NOT_FOUND",
      "message": "No instance exist."
    }
    

Response Body: [ "Orchestrator" ]

Response Schema: List of distinct service names.

 Responses
HTTP Status Description
200 Returned all distinct service names.
401 Invalid token
404 No instance is currently available with broker service.

Recovery APIs

The following recovery APIs are designed to re-trigger requests for pending tasks.

  • /v1/notifications/pending

  • /v1/notifications/re-send

  • /v1/notifications/re-send/all

For more information about recovery APIs, see the TIBCO® Order Management Web Services Guide.

Alternatively, re-trigger functions can be performed through the OMS UI. For more information, see Pending Tasks.

Instance Status Management

ActiveInstanceMonitor

In scenarios where an active instance experiences an unexpected shutdown before updating its health status with the Broker service, the database might retain the instance's status as "Active."

The primary goal of the ActiveInstanceMonitor is to detect such instances that might be inactive due to an unforeseen disruption. This monitor initiates the transfer of any pending southbound replies from the affected instance to other operational instances, ensuring the continuity of service and preventing interruptions in order processing.

 Configuration Used
Property Name Purpose
activeInstanceMonitorInterval Time interval, measured in minutes, at which ActiveInstanceMonitor operates or runs.
activeInstanceMonitorThreshold The duration parameter used by ActiveInstanceMonitor to identify and declare inactive instances based on the timestamp comparison (LastUpdatedTimestamp) against the defined threshold.

The values of activeInstanceMonitorInterval and activeInstanceMonitorThreshold are set in consideration of the brokerPingInterval (available in ConfigValues_Orchestrator, used by Orchestrator). These values play a role in determining how often the ActiveInstanceMonitor evaluates the last updated timestamp and subsequently marks instances as inactive.

activeInstanceMonitorInterval is chosen to ensure that the ActiveInstanceMonitor runs frequently enough to promptly identify instances that might have missed their health status updates. It must be set at a frequency that allows for timely checks without being overly burdensome on the system.

activeInstanceMonitorThreshold must be configured in relation to the brokerPingInterval. The threshold value defines the maximum time allowed before considering an instance as inactive. It's essential to set it considering the expected frequency of instance health status pings (brokerPingInterval). The threshold must ideally be longer than the brokerPingInterval to allow for a margin of error due to delays or missed pings.

For example, if brokerPingInterval is set at 5 minutes, it might be reasonable to configure activeInstanceMonitorInterval slightly longer than that, such as 7-10 minutes, and then set the activeInstanceMonitorThreshold to a value greater than the brokerPingInterval, for instance, at 15-20 minutes. This setup allows the ActiveInstanceMonitor to run more frequently than the pings to ensure quick detection of inactive instances without marking instances inactive due to minor communication delays.

Behavior

  • The ActiveInstanceMonitor operates as a time-based scheduler, executing at intervals set by activeInstanceMonitorInterval.

  • It calculates the `lastUpdatedTimestampThreshold` using the formula System.currentTimeMillis() - (long) activeInstanceMonitorThreshold.

  • With this threshold, the monitor identifies instances that haven't been updated within the specified duration—specifically, instances whose lastUpdatedTimestamp is less than or equal to the lastUpdatedTimestampThreshold. Instances that meet this criterion are marked as inactive, indicating potential disruptions or a failure to report their health status.

  • For each instance that meets this criterion, the monitor sets the instance status to Inactive in the database and triggers the dispatch of an OrderEventTransferTask to EMS.

InactiveInstanceMonitor

When an instance becomes inactive, there might be situations where southbound systems send replies for orders previously processed by that instance. Since the inactive instance is no longer operational, these messages accumulate on EMS queues, affecting order processing.

The primary goal of the InactiveInstanceMonitor is to relocate all such pending messages to an active instance. This action ensures that order processing continues uninterrupted, despite the inactivity of some instances.

 Configuration Used
Property Name Purpose
inactiveInstanceMonitorCronExpression A Cron expression that determines the execution schedule of InactiveInstanceMonitor.

Behavior

  • This monitor fetches instances that are in inactive status.

  • For all such instances, OrderEventTransferTask is dispatched to EMS.

Instance Failure Mechanism

OrderEventTransferMonitor

The OrderEventTransferMonitor is designed to transfer all pending events in the Event Management System (EMS) queues associated with instances that are not in an active status. This is achieved by creating a separate thread for each queue. Additionally, it updates the originator (specified in the Java Message Service (JMS) header) for all messages present at that specific time.

The following queues are affected:

  • tibco.aff.orchestrator.planItem.execute.reply

  • tibco.aff.orchestrator.planItem.suspend.reply

  • tibco.aff.orchestrator.provider.planItem.failed.reply

  • tibco.aff.orchestrator.provider.order.opd.reply

  • tibco.aff.orchestrator.planItem.milestone.notify.request

  • tibco.aff.orchestrator.provider.order.prequal.failed.reply

  • tibco.aff.orchestrator.provider.order.feasibility.reply

Input

{
	"instanceId": "",
	"ipAddress": "",
	"port": ""
}
 Input Schema
Parameter Description
instanceId An exclusive identifier associated with the instance
ipAddress IP address of the instance
port Port of the instance
 Configuration Used
Property Name Purpose
emsServerUrl EMS Server URL
emsServerUsername EMS Server User Name
emsServerPassword EMS Server Password
timeoutReceiveCalls Ems message receive timeout in milliseconds
orderEventTransferTaskQueue Queue on OrderEventTransfer task would be published internally by Broker Service
orderEventTransferTaskListenerCount Number of concurrent listener on orderEventTransferTaskQueue

Behavior

  • Order event initiation occurs under the following circumstances:

    • An active node has not updated its status within a specified duration.

    • An instance remains in an inactive status.

  • Verification of instance availability is performed with the Broker for a given instanceId. This task is skipped if the instance is not found.

  • If the instance status is not Active, the order event transfer process is initiated for all relevant queues.

  • Individual threads are created for each relevant queue.

  • These threads establish connections with the EMS Server and begin monitoring the designated queue.

  • After identifying a message, the thread reassigns the order to a new instance. The Broker updates the existing JMS Header value named Originator with the new value = new InstanceId. For more information, see the Re-assigning Order to an Instance section.

  • The Broker updates the instanceId associated with the order in the Order_data table.

  • The message, now carrying the updated Originator header, is dispatched back to the same queue for further processing.

Re-assigning Order to an Instance

The primary objective of this task is to reallocate orders previously processed by an inactive instance to an instance currently marked as Active.

 Input
Parameter Description
orderId An exclusive identifier associated with the order.
currentInstanceId An exclusive identifier associated with the instance that was processing this order and is currently not in Active state.

Behavior

  • Fetch the instanceId assigned to a given orderId.

  • If the instanceId matches the currentInstanceId, reassign the order to a new instance.

  • If the instanceId does not match the currentInstanceId, it implies that the order might already be assigned to another new instance. Subsequently, the task checks the status of this new instance. If the new instance is found to be Inactive, the broker reassigns the order to an alternate new instance.

Logic to compute new InstanceId

  • Retrieve all active instances available within the Broker Service.

  • It assesses the workload managed by each instance using the following query.

    select node_id, count(*)
    from plan_item_data pid
             inner join order_data od on pid.orderid = od.orderid
    where od.node_id is not null
      and od.node_id in :NODE_ID_LIST
      and od.status NOT in ('COMPLETE', 'CANCELLED', 'WITHDRAWN')
      and pid.status not in ('COMPLETE', 'CANCELLED')
    group by node_id
    
  • The order is assigned to the instance handling the least workload among the available active instances.

Message routing On EMS

 Configuration Used
Property Name Description Default Value
orchestratorInboundQueue Queue on which the Orchestrator listens for all southbound replies that contain an originator header. tibco.aff.orchestrator.inbound.queue
orchestratorInboundNoOriginatorQueue Queue on which the Broker Service listens for all southbound replies that do not contain an originator header. tibco.aff.orchestrator.inbound.no.originator.queue

Bridges at the EMS level are introduced to optimize message handling between the southbound service and the Orchestrator. These bridges facilitate the routing of replies based on the presence of an originator header in the message.

Southbound Message Routing

When the southbound service sends a reply to the reply queue, the message is bridged from the source queue to a specific target queue based on a selector.

  • With Originator Header: If the southbound reply contains the originator header, the message is redirected to the orchestratorInboundQueue.

  • Without Originator Header: If the southbound reply does not contain the originator header, the message is redirected to the orchestratorInboundNoOriginatorQueue.

When the southbound service sends a Plan Item Execute Reply to the tibco.aff.orchestrator.planItem.execute.reply queue, the EMS routes this message to either orchestratorInboundQueue or orchestratorInboundNoOriginatorQueue based on the presence of the originator header in the message.

System Adaptation and Customer Impact

This bridging mechanism ensures that the system adapts to these changes without requiring any modifications on the customer’s end. The orchestration and broker services handle the messages as follows:

  • Orchestrator Service: Listens for messages on orchestratorInboundQueue.

  • Broker Service: Listens for messages on orchestratorInboundNoOriginatorQueue.

Advantages of the New Approach

This approach provides significant advantages over a model where microservices have a fixed number of listeners on each queue:

  • Thread Utilization: Fixed listeners on multiple queues can lead to blocked threads when there are no messages, wasting resources.

  • Load Balancing: With the new approach, a fixed number of threads on a single queue process all incoming messages, ensuring efficient load balancing and optimal resource utilization.

By consolidating the message processing to a single queue with intelligent routing based on message headers, the service maintains high efficiency and flexibility in handling southbound messages. This design minimizes thread blocking and enhances the system's ability to manage varying message loads effectively.

Listener for Southbound Reply With no Originator

When Orchestrator dispatches requests to Southbound systems, it includes its InstanceId in a header named 'Originator'.

This setup is designed for the Southbound systems to respond with a message containing the same 'Originator' header. Each Orchestrator instance is configured to listen solely to messages associated with its InstanceId, using a specific selector, 'Originator=<<InstanceId>>'.

However, there are scenarios where a Southbound system replies without this expected header. In such cases, the Broker service intervenes by monitoring messages lacking the 'Originator' header using a selector, 'Originator IS NULL'. The Broker service then directs these messages to the instance processing the order.

If this instance is inactive in the Broker database, the Broker service assigns a new instance for handling this order. For more information, see the Re-assigning Order to an Instance section.

 Configuration Used
Property Name Purpose
orchestratorInboundNoOriginatorQueue Queue from which the broker service listens for messages that do not contain an originator header.
orchestratorInboundNoOriginatorReceiverCount Number of concurrent consumers on No originator inbound queue
orchestratorInboundNoOriginatorRetryCount Number of retries in case of any exception
orchestratorInboundNoOriginatorRetryInterval Delay in Milliseconds between consecutive retries
orchestratorInboundNoOriginatorDeadQueue Dead queue to redirect the failure messages
orchestratorInboundQueue Queue from which the Orchestrator listens for Southbound messages. The broker service redirects messages to this queue for processing by the Orchestrator.

Behavior

  • The Broker service operates by monitoring individual southbound response queues.

  • After receiving a message, it extracts specific order identifiers such as orderId, orderRef, and planId (optionally), to search for the corresponding order in the order database.

  • When the order is located in the database, the Broker service computes the instance id for that particular order, following the process detailed in the "Re-assigning order to an instance" section.

  • The Broker service forwards the message back onto the same queue, adding a header named 'Originator' with the value set as instanceId.

Error Handling

In the event of an exception during message processing, the system initiates a retry mechanism. The process is retried for a specified count of attempts, defined as noOrignatorRetryCount, with a delay between each attempt, set as noOriginatorRetryInterval.

Recovery Notification Processing

 Configuration Used
Property Name Purpose
recoveryNotificationQueue The queue to listen to the recovery notification sent by Orchestrator.
recoveryNotificationDeadQueue Dead Queue to send the unprocessable recovery notification.
recoveryNotificationReceiverCount Number of listener to listen to the recovery notification.
recoveryNotificationRetryCount Number of retries in case of failure.
recoveryNotificationRetryInterval Interval in milliseconds between each retry.

The recovery notifications are managed by the Broker service as follows:

  • Notification Dispatch: The Orchestrator dispatches a notification to the queue specified by the recoveryNotificationQueue property.
  • Broker Service Responsibility: The Broker service listens to these notifications and save them in the notification table, tracking the status and ensuring proper handling of Southbound requests and replies.

Time Dependency Monitor

The purpose of the TimeDependencyMonitor class is to ensure the consistency and correctness of time dependencies associated with orders in the system. It performs periodic checks to address orphan time dependencies, which are those with null instance IDs, assigning them to active owners based on predefined rules.

 Configuration Used
Property Name Purpose
timeDependencyMonitorInterval Fixed interval in minutes to monitor time dependency with null instance id.

Behavior

  • This job fetches all the orphan time dependencies from the database and groups them by order ID.

  • For each order ID, it checks if the order has an owner assigned to it.

    • If an order has a valid instance ID (for example, the instance is active), the scheduler updates the time dependency with that instance ID.

    • If an order's instance is inactive or does not exist, the scheduler computes a new instance ID and assigns it to the time dependency and order.