Design and Implementation
This section describes the design of the Broker service and its implementation.
Instance Status Descriptions
During the instance registration and deregistration processes, instances undergo the following statuses:
-
Pending: Indicates that an instance is registered but verification is pending to confirm whether the instance has started successfully without errors.
-
Active: Signifies that an instance is registered and currently active.
-
Inactive: Indicates that an instance is not currently active. An instance is marked as inactive by the
ActiveInstanceMonitor
flag when it has not communicated or pinged back for an extended period.
Database Details
The following table shows the database details for Broker service.
Name | Description | Data Type | |
---|---|---|---|
Oracle DB | PostgreSQL DB | ||
instance_id | Unique identifier for the registered instance | VARCHAR2(255) | character varying(255) |
ip_address | IP address associated with a specific instance | VARCHAR2(250) | character varying(250) |
port | Port number associated with a particular service or endpoint, specifies the network port used for communication. | VARCHAR2(20) | character varying(20) |
status | Current operational status or state of a system component or service, aiding in monitoring and management by indicating whether it is active, inactive | NUMBER(10, 0) | numeric(10, 0) |
service_name | The name or identifier of a specific service, aiding in the categorization and organization of services within the system. | VARCHAR2(50) | character varying(50) |
last_updated_timestamp | Last timestamp for instance status updated | NUMBER(20, 0) | numeric(20, 0) |
The database scripts are present at the following locations:
PostgreSQL database: <OM_HOME>/db/dbscripts/postgreSQL/broker
Oracle database: <OM_HOME>/db/dbscripts/oracle/broker
Configuration Value Details
Properties File Update
Before the server startup, the Broker service uses the configurator component to retrieve essential configuration properties from the application.properties
file, which ensures a seamless startup process.
Property Name | Property Value | Description |
---|---|---|
server.port | 9105 | Port at which the broker service starts. |
configuratorServiceUrl | http://localhost:9090 | Configurator base URL to connect and download configuration properties and configuration files. |
configuratorServiceRetryCount | 5 | Number of retries to perform in case of request failure to configurator. |
configuratorServiceRetryDuration | 5 | Delay between each retry, in seconds. |
configuratorTrustStoreAbsoluteFilePath | C:/Users/cacert | Trust store file path if configurator is on HTTPS. |
configuratorTrustStorePassword | tibco123 | Trust store password if configurator is on HTTPS. |
configuratorTrustStoreType | jks | Trust store type if configurator is on HTTPS. |
security.key | ENC(nSa0k6lmjPPN8ZA5SO6BpQ==) | Security key to securely connect to Configurator. The default value is encrypted of 1t1s@asy. |
Update Broker Service configuration values
The configurator must have certain configuration properties available. Before the server starts, you must upload these configuration properties to the configurator. These properties are available in the $OM_HOME/seed-data/app-properties/ConfigValues_BrokerService.json
file.
Catalog Client Configuration
When the Broker service triggers pending southbound notifications, it populates the plan item details in some of those notifications. To achieve this, the Broker service connects to the Catalog Service and fetches the latest plan fragment details associated with the plan item, then includes these details in the southbound notifications.
Conversely, the Orchestrator fetches these notifications by directly connecting to the catalog database and caching the details in its memory. Unlike the Orchestrator, the Broker service retrieves data through REST calls to the Catalog service and does not cache the responses.
Property Name | Property Value | Description |
---|---|---|
catalogServiceBaseUrl | http://localhost:9092 | Catalog Service base URL. |
catalogTrustStorePassword | tibco123 | Trust store password if Catalog is on HTTPS. |
catalogTrustStoreType | jks | Trust Store type if Catalog is on HTTPS. |
catalogTrustStoreFileName | cacert | Trust Store certificate file name. This is required if the catalog is running on HTTPS. The Broker Service downloads this file from the configurator. |
Broker Database Configuration
The Broker Service manages instance life-cycles by storing instance information in a database. For more information, see the Instance Management APIs.
Property Name | Property Value | Description |
---|---|---|
brokerDsUrl | jdbc:postgresql://localhost:5432/brokerdbll? currentSchema=brokerschemall |
JDBC URL of the broker database. |
brokerDsUsername | brokeruserll | User name of the broker database. |
brokerDsPassword | brokeruserll | Password of the broker database. |
brokerDsInitializeSize | 2 | Number of connections that are to be established when the connection pool is started. |
brokerDsMaxIdle | 11 | Maximum number of connections that must be kept in the idle pool. |
brokerDsMaxActive | 12 | Maximum number of active connections that can be allocated from this pool at the same time. |
brokerDsMaxWait | 1000 | Maximum number of milliseconds that the pool waits when there are no available connections. |
brokerDsTestOnBorrow | false | Enable connection validation before being borrowed from the pool. |
brokerDsValidationInterval | 5000 | Data source validation interval in milliseconds. |
brokerDsTestWhileIdle | true | Enable connection validation while idle in connection pool. |
brokerDsTimeBetweenEvictionRunsMillis | 5000 | Minimum amount of time in milliseconds an object must sit idle in the pool before it is eligible for eviction. |
brokerDsMinEvictableIdleTimeMillis | 5000 | Minimum amount of time in milliseconds an object might sit idle in the pool. |
brokerDsNumTestsPerEvictionRun | 5 | Maximum number of connections to examine during each evictor run. |
brokerDsDefaultAutoCommit | false | Enable auto commit after each transaction. |
brokerDsRollbackOnReturn | false | Enable transaction rollback when connection is returned back to the pool. |
brokerDsCommitOnReturn | false | Enable transaction commit when connection is returned back to the pool. |
brokerDsCustomProperty | Pooled Database Custom Property. |
Instance Management Configuration
The Broker service includes job schedulers that monitor instance life-cycles. The following properties are required for these jobs. For more information, see Instance Status Management and Instance Failure Mechanism.
Property Name | Property Value | Description |
---|---|---|
activeInstanceMonitorInterval | 20 | Time interval, measured in minutes, at which ActiveInstanceMonitor operates or runs. |
activeInstanceMonitorThreshold | 5 | The duration parameter used by ActiveInstanceMonitor to identify and declare inactive instances based on the timestamp comparison (LastUpdatedTimestamp) against the defined threshold. |
inactiveInstanceMonitorCronExpression | 0 */20 * * * ? | A Cron expression that determines the execution schedule of InactiveInstanceMonitor. |
orderEventTransferTaskQueue | com.tibco.broker.order.event.task.queue | Queue on OrderEventTransfer task would be published internally by Broker Service. |
orderEventTransferTaskListenerCount | 10 | Number of concurrent listener on the orderEventTransferTaskQueue. |
orderEventTransferTaskRetryCount | 5 | Retry count for orderEventTransferTask. |
orderEventTransferTaskRetryInterval | 1 | Retry interval between orderEventTransferTask . |
timeDependencyMonitorInterval | 1 | Fixed interval in minutes to monitor time dependency with null instance id. |
No Originator Listener Configuration
The Broker maintains a connection to the EMS server to listen for southbound replies. This includes listening for replies from currently inactive instances and those lacking the required originator header. Additionally, to ensure the Broker service can recover requests sent to the southbound system, it must be configured with the appropriate queues.
For more information, see the following topics:
Property Name | Property Value | Description |
---|---|---|
orchestratorInboundNoOriginatorQueue | tibco.aff.orchestrator.inbound.no.originator.queue | Queue on which Orchestrator listens for incoming replies from the southbound system. |
orchestratorInboundNoOriginatorDeadQueue | tibco.aff.orchestrator.inbound.no.originator.dead.queue | Dead Queue on which Orchestrator sends the unprocessable messages. |
orchestratorInboundNoOriginatorReceiverCount | 5 | Number of concurrent consumers on each southbound response queue. |
orchestratorInboundNoOriginatorRetryCount | 5 | Number of retries in case of any exception. |
orchestratorInboundNoOriginatorRetryInterval | 5000 | Delay in Milliseconds between consecutive retries. |
Recovery Notification Configuration
Broker service requires the following properties to process recovery notifications. For more information, see Recovery Notification Processing.
Property Name | Property Value | Description |
---|---|---|
recoveryNotificationQueue | tibco.aff.orchestrator.order.event.notification.queue | The queue to listen to the recovery notification sent by Orchestrator. |
recoveryNotificationDeadQueue | tibco.aff.orchestrator.order.event.notification.queue.dead | Dead Queue to send the unprocessable recovery notification. |
recoveryNotificationReceiverCount | 5 | Number of listeners to listen to the recovery notification. |
recoveryNotificationRetryCount | 5 | Number of retries in case of failure. |
recoveryNotificationRetryInterval | 5000 | Interval in milliseconds between each retry. |
Update Orchestrator Service Configuration Values
You need to update the Orchestrator service configuration values. The properties are available in the $OM_HOME/seed-data/app-properties/ConfigValues_OrchService.json
file.
Order Database Configuration
Broker service interacts with the order database to carry out the following functions:
-
Re-trigger pending notifications.
-
Determine a new instance ID for an order in situations where,
-
The owner instance becomes inactive.
-
Southbound replies do not include the required originator header.
For more information, see the following topics:
-
Property Name | Property Value | Description |
---|---|---|
orderDsUrl |
jdbc:postgresql://localhost:5432/ orderdbllhf7?currentSchema=orderschemallhf7 |
JDBC URL of the broker database. |
orderDsUsername | orderuserll | User name of the broker database. |
orderDsPassword | orderuserll | Password of the broker database. |
orderDsInitializeSize | 2 | Number of connections that are to be established when the connection pool is started. |
orderDsMaxIdle | 11 | Maximum number of connections that must be kept in the idle pool. |
orderDsMaxActive | 12 | Maximum number of active connections that can be allocated from this pool at the same time. |
orderDsMaxWait | 1000 | Maximum number of milliseconds that the pool waits when there are no available connections. |
orderDsTestOnBorrow | false | Enable connection validation before being borrowed from the pool. |
orderDsValidationInterval | 5000 | Data source validation interval in milliseconds. |
orderDsTestWhileIdle | true | Enable connection validation while idle in connection pool. |
orderDsTimeBetweenEvictionRunsMillis | 5000 | Minimum amount of time in milliseconds an object must sit idle in the pool before it is eligible for eviction. |
orderDsMinEvictableIdleTimeMillis | 5000 | Minimum amount of time in milliseconds an object might sit idle in the pool. |
orderDsNumTestsPerEvictionRun | 5 | Maximum number of connections to examine during each evictor run. |
orderDsDefaultAutoCommit | false | Enable auto commit after each transaction. |
Common Database Configuration
Property Name | Property Value | Description |
---|---|---|
datasourceDriverClassName | org.postgresql.Driver | Data Source Driver Class Name. |
datasourceValidationQuery | SELECT 1 | SQL query that is used to validate connections. |
hibernateDialect | org.hibernate.dialect.PostgreSQLDialect | Hibernate Dialect. |
hibernateShowSql | false | Hibernate Show SQL. |
EMS Configuration
The Broker stays connected to the EMS Server to listen for southbound replies, especially for instances that are inactive now and for replies lacking the required originator header.
Property Name | Property Value | Description |
---|---|---|
emsServerURL | tcp://localhost:7222 | EMS Server URL |
emsServerUsername | admin | EMS Server User Name |
emsServerPassword | admin | EMS Server Password |
jndiConnectionFactory | GenericConnectionFactory | JNDI Connection Factory Name |
tibjmsNamingSecurityProtocol | SSL configuration value, to be set to 'ssl'. | |
tibjmsNamingSslEnableVerifyHost | Set to 'false' to disable host verification in SSL. | |
initialContextFactory | com.tibco.tibjms.naming. TibjmsInitialContextFactory |
Name of the Initial Context Factory. |
jmsSessionTransacted | true | Determines if JMS sessions are transacted. Set to 'true'. |
timeoutReceiveCalls | 10000 | Specifies the timeout for acknowledging EMS messages, in milliseconds. |
Messaging Queues
The Broker stays connected to the EMS Server to listen for Southbound replies, especially for instances that are inactive now and for replies lacking the required originator header. Additionally, to recover requests sent to the Southbound system, the Broker requires proper queues configuration.
For more information, see the following topics:
Property Name | Property Value | Description |
---|---|---|
orchestratorInboundQueue | tibco.aff.orchestrator.inbound.queue |
The Broker service monitors the Orchestrator inbound queue as part of the Instance Failure Mechanism to manage messages in the event of an instance failure. |
Exposed REST Endpoints
The instance management APIs are as follows:
Registration of a new instance
URI: v1/instance/register
Request Body:
{ "instanceId": "string", "serviceName": "string", "ipAddress": "string", "port": 0 }
Parameter | Description |
---|---|
serviceName | Service name of the instance. Currently only Orchestrator is supported. |
ipAddress | IP address of the instance. |
port | Port of the instance. |
instanceId | Id of the instance. |
Description
This API handles instance registration requests. After receiving a registration request, the Broker service uses a database sequence to create an instanceId
. Subsequently, it performs validations before inserting a new record in the instance_ledger
table.
Importantly, during registration, the Broker service does not immediately verify the instance's health status.
The initial status of the selected instance remains Pending
. The instance status is updated to Active
or Inactive
based on the health assessment during the ping operation.
On successful registration, an Instance Status Change Notification is dispatched.
If a registration request is received for an already registered IP address and port, the Broker service returns the existing instance ID. This ensures that all pending messages on the EMS can be processed by the same instance that was restarted.
Request Validations and corresponding error responses
-
IP Address cannot be null or empty string.
{ "status": "BAD_REQUEST", "message": "Ip Address cannot be null or empty" }
-
Port cannot be less than or equal to zero
{ "status": "BAD_REQUEST", "message": "Port cannot be less than or equals to Zero" }
-
Service Name cannot be null or empty string
{ "status": "BAD_REQUEST", "message": "Service name cannot be null or empty" }
-
Service Name cannot be anything other than Orchestrator
{ "status": "BAD_REQUEST", "message": "Invalid service name {serviceName}" }
Response Body
{ "instanceId": "string", "serviceName": "string", "ipAddress": "string", "port": int, "status": "string" }
Parameter | Description |
---|---|
instanceId | A unique identification of the instance about to be registered. |
serviceName | Service name of the instance. Currently only Orchestrator is supported. |
ipAddress | IP address of the instance. |
port | Port of the instance. |
status |
|
HTTP Status | Description |
---|---|
201 | New Instance is registered successfully. |
200 | Instance with ipAddress and port already existed with inactive status. Updated the status to Pending . |
202 | Instance with ipAddress and port already existed with active status.
No operation is performed by Broker. |
401 | Invalid token |
400 |
|
Unregistration of an existing instance
URI: v1/instance/un-register/{instance_id}
Path Variable:
instance_id
(Instance Id): A unique identification of the instance about to be unregistered. Accepted values can be any non null or non blank string.
Description
This asynchronous operation accepts the instance_id
as a path variable and deregisters an instance from the Broker service. The deregistration process does not immediately remove the instance. The status of the instance is updated to Inactive
.
Request Validations and corresponding error responses
-
Instance Id cannot be null or empty string
{ "status": "BAD_REQUEST", "message": "Instance Id cannot null or empty" }
-
Instance with id {{instance_id}} is not present
{ "status": "BAD_REQUEST", "message": "Instance with id {{instance_id}} is not present" }
HTTP Status | Description |
---|---|
202 | Request accepted |
401 | Invalid token |
400 | Instance Id cannot null or empty |
404 | Instance with id {{instance_id}} is not present |
Instance Ping Operation
URI: v1/instance/ping
Request Body
{ "instanceId": "string", "status": "string", "inactiveComponentList": [ "string" ] }
Parameter | Description |
---|---|
instanceId | A unique identification of the instance |
status |
Health status of the instance. The following values are Allowed:
|
inactiveComponentList | List of components within the instance that do not currently exhibit an active health status. |
Description
This API allows instances to regularly update their health status by communicating with the Broker Service. Instances compute their health status periodically and send this data to the Broker Service using the REST API.
After receiving these updates, the Broker Service processes and updates the instance's status in the database. If an instance reports an Up
status, it is labeled as Active
; otherwise, it is marked as Inactive
.
In cases where an instance is Inactive
, the system triggers an Order Event Transfer
task to EMS for further handling.
Request Validations and corresponding error responses
-
Instance Id cannot be null or empty string
{ "status": "BAD_REQUEST", "message": "Instance Id cannot null or empty" }
-
Instance with id
{{instance_id}}
is not present{ "status": "NOT_FOUND", "message": "Instance with id {{instance_id}} is not present" }
-
Status cannot be null or empty
{ "status": "BAD_REQUEST", "message": "Status cannot be null or empty" }
-
Invalid status
{ "status": "BAD_REQUEST", "message": "Invalid status. Only Up, Down, OUT_OF_SERVICE and UNKNOWN is allowed" }
HTTP Status | Description |
---|---|
200 | Request processed |
401 | Invalid token |
400 |
|
404 | Instance with id {{instance_id}} is not present. |
Instance clean up
URI: v1/instance/cleanup
Request Body: [
"string"
]
Request Schema: List of instance ids to purge from the database.
Description
In some scenarios, an instance might become inactive without being removed from the Broker database. You can use this API to manually specify a list of instance IDs that must be purged from the database.
If any instance in the provided list is not in an Inactive
status, it is not purged. However, any other instances listed is removed from the database.
Request Validations and corresponding error responses
-
Instance Id list cannot be empty
{ "status": "BAD_REQUEST", "message": "Instance Id list cannot be empty" }
-
Are any of the given instances available with broker service?
{ "status": "NOT_FOUND", "message": "None of requested instances are present" }
-
Are all the given instances still active?
{ "status": "BAD_REQUEST", "message": "All of the requested instances are still active. Unregister them first." }
HTTP Status | Description |
---|---|
200 | All requested instances are purged from the database. |
206 | Only some of the requested instances are purged from the database. This would usually occur when some instances are still active. |
401 | Invalid token |
400 |
|
404 | None of request instances are present |
Get Instance Details By Instance Id
URI: v1/instance/{instanceId}
Path Variable:
instance_id
(Instance Id): An exclusive identifier associated with the instance for which details are being sought. The accepted values can be any non null or non blank string.
Description
This API accepts an instanceId
as a path variable and returns the details of the instance that is registered with the broker service.
Request Validations and corresponding error responses
-
Instance Id cannot be null or empty string
{ "status": "BAD_REQUEST", "message": "Instance Id cannot be null or empty string" }
-
Instance with id
<<instanceId>>
is not present{ "status": "NOT_FOUND", "message": "Instance with id <<instanceId>> is not present" }
Response Body
{ "instanceId": "abc", "serviceName": "Orchestrator", "ipAddress": "0.0.0.0", "port": 9093, "status": "UP" }
Parameter | Description |
---|---|
instanceId | A unique identification of the instance about to be registered. |
serviceName | Service name of the instance. Currently only Orchestrator is supported. |
ipAddress | IP address of the instance. |
port | Port of the instance. |
status |
|
HTTP Status | Description |
---|---|
200 | Returned the instance details of the requested instance. |
401 | Invalid token |
400 | Instance Id cannot be null or empty string. |
404 |
Instance with id <<instanceId>> is not present. |
Get Instance Details By Service Name
URI: /v1/instance?serviceName={serviceName}&pageNo={pageNo}&instancesPerPage={instancesPerPage}
Parameters |
Description |
---|---|
serviceName (Service Name) |
The service name for which details of every available instance are being requested. The accepted values can be any non-null or non-blank string. |
pageNo (Page Number) |
This parameter specifies the page number of the paginated data. The accepted values can be positive integers starting from 1, where 1 represents the first page. |
instancesPerPage (Instances Per Page) |
This parameter determines the number of instances displayed on a single page of paginated data. The accepted values can be positive integers indicating the desired number of instances per page. |
Description
You can use this API endpoint to retrieve a paginated list of records from the system for a given serviceName
. You can specify the service name, page number, and the number of records to display per page using query parameters.
Request Validations and corresponding error responses
-
Service name cannot be null or empty string
{ "status": "BAD_REQUEST", "message": "Service name cannot be null or empty" }
-
No instance is available for the given service name
{ "status": "NOT_FOUND", "message": "No instances found for service name {serviceName}" }
-
Invalid Page number. Page number cannot be less than 1.
{ "status": "BAD_REQUEST", "message": "PageNo cannot be less than 1" }
-
Invalid instances per page. InstancesPerPage must be between 1 and 20.
{ "status": "BAD_REQUEST", "message": "InstancesPerPage should be between 1 and 20." }
Response Body
{ "instanceDetailsList": [ { "instanceId": "abc", "serviceName": "Orchestrator", "ipAddress": "0.0.0.0", "port": 9093, "status": "UP" } ], "totalPages": 1, "currentPage": 1, "instancesPerPage": 10 }
Parameter | Description |
---|---|
instanceDetailsList | List of the instance details |
totalPages | Total number of pages available as per instancesPerPage |
currentPage | Current Page number |
instancesPerPage | Instances per page requested by user |
Parameter | Description |
---|---|
instanceId | A unique identification of the instance about to be registered. |
serviceName | Service name of the instance. Currently only Orchestrator is supported. |
ipAddress | IP address of the instance |
port | Port of the instance |
status |
|
HTTP Status | Description |
---|---|
200 | Returns all instance details for all the instances registered with requested service name. |
401 | Invalid token |
400 |
|
404 | No instances found for service name {serviceName} |
Get Instance Details of all instances
URI: v1/instance/all?pageNo={pageNo}&instancesPerPage={instancesPerPage}
Parameters |
Description |
---|---|
pageNo (Page Number) |
This parameter specifies the page number of the paginated data. The accepted values can be positive integers starting from 1, where 1 represents the first page. |
instancesPerPage (Instances Per Page) |
This parameter determines the number of instances displayed on a single page of paginated data. The accepted values can be positive integers indicating the desired number of instances per page. |
Description
You can use this API endpoint to retrieve a paginated list of records from the system. Also, you can specify the page number and the number of records to display per page using query parameters.
Request Validations and corresponding error responses
-
Invalid Page number. Page number cannot be less than 1
{ "status": "BAD_REQUEST", "message": "PageNo cannot be less than 1" }
-
Invalid instances per page. InstancesPerPage must be between 1 and 20.
{ "status": "BAD_REQUEST", "message": "InstancesPerPage should be between 1 and 20." }
Response Body
{ "instanceDetailsList": [ { "instanceId": "abc", "serviceName": "Orchestrator", "ipAddress": "0.0.0.0", "port": 9093, "status": "UP" } ], "totalPages": 1, "currentPage": 1, "instancesPerPage": 10 }
Parameter | Description |
---|---|
instanceDetailsList | List of the instance details |
totalPages | Total number of pages available as per instancesPerPage. |
currentPage | Current Page number |
instancesPerPage | Instances per page requested by user |
Parameter | Description |
---|---|
instanceId | A unique identification of the instance about to be registered. |
serviceName | Service name of the instance. Currently only Orchestrator is supported. |
ipAddress | IP address of the instance |
port | Port of the instance |
status |
|
HTTP Status | Description |
---|---|
200 | Returned all instance details for all the instances registered with requested service name. |
401 | Invalid token |
400 |
|
404 | No instances available |
Get all services
URI: /v1/instance/services
Description
This API provides a distinct list of service names for which instances are currently available within the Broker service.
Request Validations and corresponding error responses
-
No instance is available
{ "status": "NOT_FOUND", "message": "No instance exist." }
Response Body: [
"Orchestrator"
]
Response Schema: List of distinct service names.
HTTP Status | Description |
---|---|
200 | Returned all distinct service names. |
401 | Invalid token |
404 | No instance is currently available with broker service. |
Recovery APIs
The following recovery APIs are designed to re-trigger requests for pending tasks.
-
/v1/notifications/pending
-
/v1/notifications/re-send
-
/v1/notifications/re-send/all
For more information about recovery APIs, see the TIBCO® Order Management Web Services Guide.
Alternatively, re-trigger functions can be performed through the OMS UI. For more information, see Pending Tasks.
Instance Status Management
ActiveInstanceMonitor
In scenarios where an active instance experiences an unexpected shutdown before updating its health status with the Broker service, the database might retain the instance's status as "Active."
The primary goal of the ActiveInstanceMonitor is to detect such instances that might be inactive due to an unforeseen disruption. This monitor initiates the transfer of any pending southbound replies from the affected instance to other operational instances, ensuring the continuity of service and preventing interruptions in order processing.
Property Name | Purpose |
---|---|
activeInstanceMonitorInterval | Time interval, measured in minutes, at which ActiveInstanceMonitor operates or runs. |
activeInstanceMonitorThreshold | The duration parameter used by ActiveInstanceMonitor to identify and declare inactive instances based on the timestamp comparison (LastUpdatedTimestamp) against the defined threshold. |
The values of activeInstanceMonitorInterval
and activeInstanceMonitorThreshold
are set in consideration of the brokerPingInterval
(available in ConfigValues_Orchestrator
, used by Orchestrator). These values play a role in determining how often the ActiveInstanceMonitor
evaluates the last updated timestamp and subsequently marks instances as inactive.
activeInstanceMonitorInterval
is chosen to ensure that the ActiveInstanceMonitor
runs frequently enough to promptly identify instances that might have missed their health status updates. It must be set at a frequency that allows for timely checks without being overly burdensome on the system.
activeInstanceMonitorThreshold
must be configured in relation to the brokerPingInterval. The threshold value defines the maximum time allowed before considering an instance as inactive. It's essential to set it considering the expected frequency of instance health status pings (brokerPingInterval
). The threshold must ideally be longer than the brokerPingInterval
to allow for a margin of error due to delays or missed pings.
For example, if brokerPingInterval
is set at 5 minutes, it might be reasonable to configure activeInstanceMonitorInterval
slightly longer than that, such as 7-10 minutes, and then set the activeInstanceMonitorThreshold to a value greater than the brokerPingInterval
, for instance, at 15-20 minutes. This setup allows the ActiveInstanceMonitor
to run more frequently than the pings to ensure quick detection of inactive instances without marking instances inactive due to minor communication delays.
Behavior
-
The
ActiveInstanceMonitor
operates as a time-based scheduler, executing at intervals set byactiveInstanceMonitorInterval.
-
It calculates the `
lastUpdatedTimestampThreshold
` using the formulaSystem.currentTimeMillis() - (long) activeInstanceMonitorThreshold
. -
With this threshold, the monitor identifies instances that haven't been updated within the specified duration—specifically, instances whose
lastUpdatedTimestamp
is less than or equal to thelastUpdatedTimestampThreshold
. Instances that meet this criterion are marked as inactive, indicating potential disruptions or a failure to report their health status. -
For each instance that meets this criterion, the monitor sets the instance status to
Inactive
in the database and triggers the dispatch of anOrderEventTransferTask
to EMS.
InactiveInstanceMonitor
When an instance becomes inactive, there might be situations where southbound systems send replies for orders previously processed by that instance. Since the inactive instance is no longer operational, these messages accumulate on EMS queues, affecting order processing.
The primary goal of the InactiveInstanceMonitor
is to relocate all such pending messages to an active instance. This action ensures that order processing continues uninterrupted, despite the inactivity of some instances.
Property Name | Purpose |
---|---|
inactiveInstanceMonitorCronExpression | A Cron expression that determines the execution schedule of InactiveInstanceMonitor. |
Behavior
-
This monitor fetches instances that are in inactive status.
-
For all such instances, OrderEventTransferTask is dispatched to EMS.
Instance Failure Mechanism
OrderEventTransferMonitor
The OrderEventTransferMonitor
is designed to transfer all pending events in the Event Management System (EMS) queues associated with instances that are not in an active status. This is achieved by creating a separate thread for each queue. Additionally, it updates the originator (specified in the Java Message Service (JMS) header) for all messages present at that specific time.
The following queues are affected:
-
tibco.aff.orchestrator.planItem.execute.reply
-
tibco.aff.orchestrator.planItem.suspend.reply
-
tibco.aff.orchestrator.provider.planItem.failed.reply
-
tibco.aff.orchestrator.provider.order.opd.reply
-
tibco.aff.orchestrator.planItem.milestone.notify.request
-
tibco.aff.orchestrator.provider.order.prequal.failed.reply
-
tibco.aff.orchestrator.provider.order.feasibility.reply
Input
{ "instanceId": "", "ipAddress": "", "port": "" }
Parameter | Description |
---|---|
instanceId | An exclusive identifier associated with the instance |
ipAddress | IP address of the instance |
port | Port of the instance |
Property Name | Purpose |
---|---|
emsServerUrl | EMS Server URL |
emsServerUsername | EMS Server User Name |
emsServerPassword | EMS Server Password |
timeoutReceiveCalls | Ems message receive timeout in milliseconds |
orderEventTransferTaskQueue | Queue on OrderEventTransfer task would be published internally by Broker Service |
orderEventTransferTaskListenerCount | Number of concurrent listener on orderEventTransferTaskQueue |
Behavior
-
Order event initiation occurs under the following circumstances:
-
An active node has not updated its status within a specified duration.
-
An instance remains in an inactive status.
-
-
Verification of instance availability is performed with the Broker for a given
instanceId
. This task is skipped if the instance is not found. -
If the instance status is not Active, the order event transfer process is initiated for all relevant queues.
-
Individual threads are created for each relevant queue.
-
These threads establish connections with the EMS Server and begin monitoring the designated queue.
-
After identifying a message, the thread reassigns the order to a new instance. The Broker updates the existing JMS Header value named
Originator
with the new value = newInstanceId
. For more information, see the Re-assigning Order to an Instance section. -
The Broker updates the
instanceId
associated with the order in theOrder_data
table. -
The message, now carrying the updated
Originator
header, is dispatched back to the same queue for further processing.
Re-assigning Order to an Instance
The primary objective of this task is to reallocate orders previously processed by an inactive instance to an instance currently marked as Active.
Parameter | Description |
---|---|
orderId | An exclusive identifier associated with the order. |
currentInstanceId | An exclusive identifier associated with the instance that was processing this order and is currently not in Active state. |
Behavior
-
Fetch the
instanceId
assigned to a given orderId. -
If the
instanceId
matches thecurrentInstanceId
, reassign the order to a new instance. -
If the
instanceId
does not match thecurrentInstanceId
, it implies that the order might already be assigned to another new instance. Subsequently, the task checks the status of this new instance. If the new instance is found to be Inactive, the broker reassigns the order to an alternate new instance.
Logic to compute new InstanceId
-
Retrieve all active instances available within the Broker Service.
-
It assesses the workload managed by each instance using the following query.
select node_id, count(*) from plan_item_data pid inner join order_data od on pid.orderid = od.orderid where od.node_id is not null and od.node_id in :NODE_ID_LIST and od.status NOT in ('COMPLETE', 'CANCELLED', 'WITHDRAWN') and pid.status not in ('COMPLETE', 'CANCELLED') group by node_id
-
The order is assigned to the instance handling the least workload among the available active instances.
Message routing On EMS
Property Name | Description | Default Value |
---|---|---|
orchestratorInboundQueue | Queue on which the Orchestrator listens for all southbound replies that contain an originator header. | tibco.aff.orchestrator.inbound.queue |
orchestratorInboundNoOriginatorQueue | Queue on which the Broker Service listens for all southbound replies that do not contain an originator header. | tibco.aff.orchestrator.inbound.no.originator.queue |
Bridges at the EMS level are introduced to optimize message handling between the southbound service and the Orchestrator. These bridges facilitate the routing of replies based on the presence of an originator
header in the message.
Southbound Message Routing
When the southbound service sends a reply to the reply queue, the message is bridged from the source queue to a specific target queue based on a selector.
-
With Originator Header: If the southbound reply contains the originator header, the message is redirected to the
orchestratorInboundQueue
. -
Without Originator Header: If the southbound reply does not contain the originator header, the message is redirected to the
orchestratorInboundNoOriginatorQueue
.
When the southbound service sends a Plan Item Execute Reply to the tibco.aff.orchestrator.planItem.execute.reply
queue, the EMS routes this message to either orchestratorInboundQueue
or orchestratorInboundNoOriginatorQueue
based on the presence of the originator header in the message.
System Adaptation and Customer Impact
This bridging mechanism ensures that the system adapts to these changes without requiring any modifications on the customer’s end. The orchestration and broker services handle the messages as follows:
-
Orchestrator Service: Listens for messages on orchestratorInboundQueue.
-
Broker Service: Listens for messages on orchestratorInboundNoOriginatorQueue.
Advantages of the New Approach
This approach provides significant advantages over a model where microservices have a fixed number of listeners on each queue:
-
Thread Utilization: Fixed listeners on multiple queues can lead to blocked threads when there are no messages, wasting resources.
-
Load Balancing: With the new approach, a fixed number of threads on a single queue process all incoming messages, ensuring efficient load balancing and optimal resource utilization.
By consolidating the message processing to a single queue with intelligent routing based on message headers, the service maintains high efficiency and flexibility in handling southbound messages. This design minimizes thread blocking and enhances the system's ability to manage varying message loads effectively.
Listener for Southbound Reply With no Originator
When Orchestrator dispatches requests to Southbound systems, it includes its InstanceId in a header named 'Originator'.
This setup is designed for the Southbound systems to respond with a message containing the same 'Originator' header. Each Orchestrator instance is configured to listen solely to messages associated with its InstanceId, using a specific selector, 'Originator=<<InstanceId>>'.
However, there are scenarios where a Southbound system replies without this expected header. In such cases, the Broker service intervenes by monitoring messages lacking the 'Originator' header using a selector, 'Originator IS NULL'. The Broker service then directs these messages to the instance processing the order.
If this instance is inactive in the Broker database, the Broker service assigns a new instance for handling this order. For more information, see the Re-assigning Order to an Instance section.
Property Name | Purpose |
---|---|
orchestratorInboundNoOriginatorQueue | Queue from which the broker service listens for messages that do not contain an originator header. |
orchestratorInboundNoOriginatorReceiverCount | Number of concurrent consumers on No originator inbound queue |
orchestratorInboundNoOriginatorRetryCount | Number of retries in case of any exception |
orchestratorInboundNoOriginatorRetryInterval | Delay in Milliseconds between consecutive retries |
orchestratorInboundNoOriginatorDeadQueue | Dead queue to redirect the failure messages |
orchestratorInboundQueue | Queue from which the Orchestrator listens for Southbound messages. The broker service redirects messages to this queue for processing by the Orchestrator. |
Behavior
-
The Broker service operates by monitoring individual southbound response queues.
-
After receiving a message, it extracts specific order identifiers such as orderId, orderRef, and planId (optionally), to search for the corresponding order in the order database.
-
When the order is located in the database, the Broker service computes the instance id for that particular order, following the process detailed in the "Re-assigning order to an instance" section.
-
The Broker service forwards the message back onto the same queue, adding a header named 'Originator' with the value set as
instanceId
.
Error Handling
In the event of an exception during message processing, the system initiates a retry mechanism. The process is retried for a specified count of attempts, defined as noOrignatorRetryCount
, with a delay between each attempt, set as noOriginatorRetryInterval
.
Recovery Notification Processing
Property Name | Purpose |
---|---|
recoveryNotificationQueue | The queue to listen to the recovery notification sent by Orchestrator. |
recoveryNotificationDeadQueue | Dead Queue to send the unprocessable recovery notification. |
recoveryNotificationReceiverCount | Number of listener to listen to the recovery notification. |
recoveryNotificationRetryCount | Number of retries in case of failure. |
recoveryNotificationRetryInterval | Interval in milliseconds between each retry. |
The recovery notifications are managed by the Broker service as follows:
- Notification Dispatch: The Orchestrator dispatches a notification to the queue specified by the
recoveryNotificationQueue
property. - Broker Service Responsibility: The Broker service listens to these notifications and save them in the notification table, tracking the status and ensuring proper handling of Southbound requests and replies.
Time Dependency Monitor
The purpose of the TimeDependencyMonitor class is to ensure the consistency and correctness of time dependencies associated with orders in the system. It performs periodic checks to address orphan time dependencies, which are those with null instance IDs, assigning them to active owners based on predefined rules.
Property Name | Purpose |
---|---|
timeDependencyMonitorInterval | Fixed interval in minutes to monitor time dependency with null instance id. |
Behavior
-
This job fetches all the orphan time dependencies from the database and groups them by order ID.
-
For each order ID, it checks if the order has an owner assigned to it.
-
If an order has a valid instance ID (for example, the instance is active), the scheduler updates the time dependency with that instance ID.
-
If an order's instance is inactive or does not exist, the scheduler computes a new instance ID and assigns it to the time dependency and order.
-