Monitoring the TERR service using JMX

The TERR service supports JMX monitoring integration. JMX monitoring is turned off by default.

Prerequisites

You can install and use JConsole for monitoring the TERR service using JMX. JConsole is provided as part of the Java SE Development Kit. (See Using JConsole from the Oracle Java documentation site.) Alternatively, you can install and use VisualVM to monitor the TERR service using JMX.
Important: Because JMX monitoring requires connecting to the specific IP address of the node, you must create a custom configuration for each node to monitor.

Procedure

  1. Stop the TERR service.
  2. Export and edit the custom.properties, setting the following properties. 
    jmx.rmi.username: username
jmx.rmi.password: password
jmx.rmi.host: <IP address of the Node Manager running TERR service>
jmx.rmi.port: 1099
jmx.active: TRUE
    See Configuring the TERR service for detailed instructions.
  3. Start the TERR service.
  4. Check the INFO logs for the connection string.
    If the setup and connection are successful, a JMX connection string is printed to logs at the INFO level. 
    2018-06-08T21:03:11,520 | INFO  | [main] c.s.s.t.ServiceConfig: Service configured JMX Connection string: service:jmx:rmi://10.10.100.60:1099/jndi/rmi://10.10.100.60:1099/jmxrmi
    • If jmx.rmi.username, jmx.rmi.password, or jmx.rmi.host are blank, then a log message is printed indicating that the property is blank, and that the JMX connection is not created.
    • If jmx.rmi.port is blank or undefined, then the port value defaults to 1099.
    • If the jmx.rmi.host is configured incorrectly, the connection times out and the service fails to start. An error message is printed to the admin UI and the logs.
    A successful client connection is printed to logs at the DEBUG level. An unsuccessful client connection attempt due to bad or missing username or password is printed at the ERROR level.
    Note: If you are connecting to a remote host, the port must be opened in the firewall to allow the connection.
  5. Open JConsole, and in the remote process field, provide the JMX connection string provided by the logs as shown. 
    service:jmx:rmi://10.10.100.60:1099/jndi/rmi://10.10.100.60:1099/jmxrmi
  6. Provide the user name and password that you set in custom.properties.
    Note: If a message is displayed indicating the connection could not be made using SSL. Would you like to try without SSL? , then click Insecure connection.
    JConsole should now display information from the service.
  7. To view metrics specific to the TERR service, click the tab MBeans.
  8. In the left panel, expend the group labeled metrics.
    Metrics are listed, including many JVM metrics. Some of the metrics specific for TERR are as follows. 
    serviceQueueCurrentSize - total number of engines currently waiting in the queue
serviceQueueEnginesDestroyed - total number of engines destroyed after successful use
serviceQueueEnginesFailed - total number of engines that failed on startup due to configuration, environmental, or other exceptions
serviceQueueEnginesInUse - total number of engines currently executing
serviceQueueEnginesStarted - total number of successful engines started
serviceQueueEnginesStarting - total number of engines currently initializing
serviceQueueIdealSize - the ideal queue size as defined by engine.queue.size in custom.properties
serviceQueueLastPortSelected - the last port chosen for engine creation
serviceUsageBytesDownloaded - total bytes downloaded through the service
serviceUsageBytesUploaded - total bytes uploaded through the service
serviceUsageCapacity - the current capacity of the service as a percentage: current session over maximum allowed concurrent sessions
serviceUsageJobs - total number of jobs the service has created and run
serviceUsageSessions - total number of sessions the service has created
serviceUsageMillisInUse - total time spent executing successful jobs, in milliseconds