Traffic Manager Metrics

Garbage Collection

Jetty Connection/Request Thread Pool Metrics: 
{
  "time": 1554204598,
  "message": {
    "0.connectionData.0.AvgConnectionDuration": 36.404666666666664,
    "0.connectionData.0.AvgRequestsPerConnection": 1,
    "0.connectionData.0.ConnectionDurationStdDev": 0.031923032437411084,
    "0.connectionData.0.Connections": 1,
    "0.connectionData.0.ConnectorUptime": "0hours2minutes7seconds",
    "0.connectionData.0.ConnectorUptimeSec": 127,
    "0.connectionData.0.Keystore": "NonSSL",
    "0.connectionData.0.MaxConnectionDuration": 36.451,
    "0.connectionData.0.MaxConnections": 2,
    "0.connectionData.0.MaxRequestsPerConnection": 1,
    "0.connectionData.0.Port": 80,
    "0.connectionData.0.RequestsPerConnectionStdDev": 0,
    "0.connectionData.0.TotalConnectionDuration": "0hours1minute49seconds",
    "0.connectionData.0.TotalConnectionDurationSec": 109,
    "0.connectionData.0.TotalConnections": 4,
    "0.connectionData.0.TotalRequests": 3,
    "0.connectionData.1.AvgConnectionDuration": "NaN",
    "0.connectionData.1.AvgRequestsPerConnection": "NaN",
    "0.connectionData.1.ConnectionDurationStdDev": 0,
    "0.connectionData.1.Connections": 0,
    "0.connectionData.1.ConnectorUptime": "0hours2minutes7seconds",
    "0.connectionData.1.ConnectorUptimeSec": 127,
    "0.connectionData.1.Keystore": "/etc/mashery-server-ssl/mashery-local.jks",
    "0.connectionData.1.MaxConnectionDuration": 0,
    "0.connectionData.1.MaxConnections": 0,
    "0.connectionData.1.MaxRequestsPerConnection": 0,
    "0.connectionData.1.Port": 443,
    "0.connectionData.1.RequestsPerConnectionStdDev": 0,
    "0.connectionData.1.TotalConnectionDuration": "0hours0minutes0seconds",
    "0.connectionData.1.TotalConnectionDurationSec": 0,
    "0.connectionData.1.TotalConnections": 0,
    "0.connectionData.1.TotalRequests": 0,
    "0.threadPoolIdleThreads": 46,
    "0.threadPoolMaxThreads": 500,
    "0.threadPoolMinThreads": 50,
    "0.threadPoolThreads": 50,
    "ingestion_time": "2019-04-02T11:29:58+00:00",
    "tag": "tml-tm.4a193af10aba.metrics.jvm.proxy.jetty_connection_pool"
  }
}
Note:
  • 0.connectionData.0 is the thread pool for plain HTTP requests (port 80).
  • 0.connectionData.1 is the thread pool for HTTPS requests (port 443).
Name Field Notes Details
Max requests threads 0.threadPoolMaxThreads Denotes the maximum concurrent requests possible on this Traffic manager instance
Inital number of threads 0.threadPoolMinThreads Threads to be created on startup
Current created threads 0.threadPoolThreads Number of threads created in the pool
Idle Threads 0.threadPoolIdleThreads Number of threads created but not utilized A high value of idle threads denotes low utilization of this instance. Check if other traffic manager instances are over-loaded
Total connections since start 0.connectionData.0.TotalConnections Number of connection requests to this instance
Avg time for requests 0.connectionData.0.AvgConnectionDuration Average time taken by the this instance to server traffic A low value denotes high throughput
Diagnostic Recipe / Alerts
Metric Field Name / Computation Notes
Efficiency ratio of 0.threadPoolThreads/0.threadPoolMaxThreads This ratio provides the utilization of this Traffic Manager. An ideal range is 75%-80%.

Low water mark - 80%

High water mark 90%

A continuous breach of high water mark indicates that this Traffic Manager instance is over loaded.

Under-utilization ratio of 0.threadPoolIdleThreads/0.threadPoolMaxThreads A very high ratio denotes that the Traffic Manager is handling fewer calls and might indicate deficiency in load balancing rules.
Max connections 0.connectionData.0.MaxConnections+0.connectionData.0.MaxConnections This is the maximum limit this Traffic Manager instance can serve requests.

Memory Utilization

Memory utilization is a combination of current usage + garbage collection (GC) metrics.

Current Usage
Current Usage: Heap 
{
  "time": 1554205856,
  "message": {
    "0.HeapMemoryUsage.committed": 39088128,
    "0.HeapMemoryUsage.init": 33554432,
    "0.HeapMemoryUsage.max": 1037959168,
    "0.HeapMemoryUsage.used": 25810688,
    "ingestion_time": "2019-04-02T11:50:56+00:00",
    "tag": "tml-tm.4a193af10aba.metrics.jvm.proxy.memory.heap"
  }
}
Current Usage: Non Heap 
{
  "time": 1554205856,
  "message": {
    "0.NonHeapMemoryUsage.committed": 58982400,
    "0.NonHeapMemoryUsage.init": 2555904,
    "0.NonHeapMemoryUsage.max": -1,
    "0.NonHeapMemoryUsage.used": 54043656,
    "ingestion_time": "2019-04-02T11:50:56+00:00",
    "tag": "tml-tm.4a193af10aba.metrics.jvm.proxy.memory.non_heap"
  }
}
Diagnostic Recipe / Alerts
Metric Field Name / Computation Unit Data Type Notes Details
Current usage 0.HeapMemoryUsage.used + 0.NonHeapMemoryUsage.used Bytes Number https://docs.oracle.com/javase/9/docs/api/java/lang/management/MemoryUsage.html
Committed memory 0.NonHeapMemoryUsage.committed + 0.HeapMemoryUsage.committed Bytes Number Guaranteed to be available to the JVM
Max memory Mostly undefined to rely on available resources of system Bytes Number Total allocated memory to the JVM
High Usage Ratio of Used/Committed N/A Number Used should always be lower than Committed. If Used>Committed, then the JVM can run Out of memory and stop. A high watermark of 80% can indicate that the JVM needs more memory

Garbage collection (GC) for the JVM is split into Young and Old GC. Old GC should be low for Traffic manager.

Old GC can also include full GC and sometimes only tenured space reclamation.

Young GC tries to recover transient, short-lived objects from Eden and Survivor space. (more details at https://docs.oracle.com/javase/8/docs/technotes/guides/management/jconsole.html#gdeao)

For traffic managers, the tenured space should reflect low usage vis-a-vis young space. The GC from the Young space should reflect greater collection than the tenured.
Note: The Young GC captures values for Tenured space and the Old GC captures values for Eden space. These can be safely ignored since they denote movement of objects between the two spaces during the life of the JVM
Garbage Collection: OLD GC 
{
  "time": 1554206878,
  "message": {
    "memoryUsageAfterGc.CodeCache.committed": 9175040,
    "memoryUsageAfterGc.CodeCache.init": 2555904,
    "memoryUsageAfterGc.CodeCache.max": 251658240,
    "memoryUsageAfterGc.CodeCache.used": 9141504,
    "memoryUsageAfterGc.CompressedClassSpace.committed": 4849664,
    "memoryUsageAfterGc.CompressedClassSpace.init": 0,
    "memoryUsageAfterGc.CompressedClassSpace.max": 1073741824,
    "memoryUsageAfterGc.CompressedClassSpace.used": 3764240,
    "memoryUsageAfterGc.EdenSpace.committed": 9109504,
    "memoryUsageAfterGc.EdenSpace.init": 8912896,
    "memoryUsageAfterGc.EdenSpace.max": 286326784,
    "memoryUsageAfterGc.EdenSpace.used": 0,
    "memoryUsageAfterGc.Metaspace.committed": 36306944,
    "memoryUsageAfterGc.Metaspace.init": 0,
    "memoryUsageAfterGc.Metaspace.max": -1,
    "memoryUsageAfterGc.Metaspace.used": 32641584,
    "memoryUsageAfterGc.SurvivorSpace.committed": 1114112,
    "memoryUsageAfterGc.SurvivorSpace.init": 1114112,
    "memoryUsageAfterGc.SurvivorSpace.max": 35782656,
    "memoryUsageAfterGc.SurvivorSpace.used": 0,
    "memoryUsageAfterGc.TenuredGen.committed": 22413312,
    "memoryUsageAfterGc.TenuredGen.init": 22413312,
    "memoryUsageAfterGc.TenuredGen.max": 715849728,
    "memoryUsageAfterGc.TenuredGen.used": 16019432,
    "ingestion_time": "2019-04-02T12:07:58+00:00",
    "tag": "tml-tm.4a193af10aba.metrics.jvm.proxy.old_gc"
  }
}{
  "time": 1554206878,
  "message": {
    "memoryUsageBeforeGc.CodeCache.committed": 9175040,
    "memoryUsageBeforeGc.CodeCache.init": 2555904,
    "memoryUsageBeforeGc.CodeCache.max": 251658240,
    "memoryUsageBeforeGc.CodeCache.used": 9141504,
    "memoryUsageBeforeGc.CompressedClassSpace.committed": 4849664,
    "memoryUsageBeforeGc.CompressedClassSpace.init": 0,
    "memoryUsageBeforeGc.CompressedClassSpace.max": 1073741824,
    "memoryUsageBeforeGc.CompressedClassSpace.used": 3764240,
    "memoryUsageBeforeGc.EdenSpace.committed": 9109504,
    "memoryUsageBeforeGc.EdenSpace.init": 8912896,
    "memoryUsageBeforeGc.EdenSpace.max": 286326784,
    "memoryUsageBeforeGc.EdenSpace.used": 7199984,
    "memoryUsageBeforeGc.Metaspace.committed": 36306944,
    "memoryUsageBeforeGc.Metaspace.init": 0,
    "memoryUsageBeforeGc.Metaspace.max": -1,
    "memoryUsageBeforeGc.Metaspace.used": 32641584,
    "memoryUsageBeforeGc.SurvivorSpace.committed": 1114112,
    "memoryUsageBeforeGc.SurvivorSpace.init": 1114112,
    "memoryUsageBeforeGc.SurvivorSpace.max": 35782656,
    "memoryUsageBeforeGc.SurvivorSpace.used": 1114112,
    "memoryUsageBeforeGc.TenuredGen.committed": 22413312,
    "memoryUsageBeforeGc.TenuredGen.init": 22413312,
    "memoryUsageBeforeGc.TenuredGen.max": 715849728,
    "memoryUsageBeforeGc.TenuredGen.used": 15496552,
    "ingestion_time": "2019-04-02T12:07:58+00:00",
    "tag": "tml-tm.4a193af10aba.metrics.jvm.proxy.old_gc"
  }
}{
  "time": 1554206878,
  "message": {
    "collectionCount": 2,
    "collectionTime": 88,
    "name": "MarkSweepCompact",
    "gcStartTime": 4300,
    "gcEndTime": 4353,
    "gcDuration": 53,
    "ingestion_time": "2019-04-02T12:07:58+00:00",
    "tag": "tml-tm.4a193af10aba.metrics.jvm.proxy.old_gc"
  }
}
Garbage Collection: Young GC 
{
  "time": 1554207478,
  "message": {
    "memoryUsageAfterGc.CodeCache.committed": 15925248,
    "memoryUsageAfterGc.CodeCache.init": 2555904,
    "memoryUsageAfterGc.CodeCache.max": 251658240,
    "memoryUsageAfterGc.CodeCache.used": 15751744,
    "memoryUsageAfterGc.CompressedClassSpace.committed": 5242880,
    "memoryUsageAfterGc.CompressedClassSpace.init": 0,
    "memoryUsageAfterGc.CompressedClassSpace.max": 1073741824,
    "memoryUsageAfterGc.CompressedClassSpace.used": 4136776,
    "memoryUsageAfterGc.EdenSpace.committed": 11010048,
    "memoryUsageAfterGc.EdenSpace.init": 8912896,
    "memoryUsageAfterGc.EdenSpace.max": 286326784,
    "memoryUsageAfterGc.EdenSpace.used": 0,
    "memoryUsageAfterGc.Metaspace.committed": 41418752,
    "memoryUsageAfterGc.Metaspace.init": 0,
    "memoryUsageAfterGc.Metaspace.max": -1,
    "memoryUsageAfterGc.Metaspace.used": 37438280,
    "memoryUsageAfterGc.SurvivorSpace.committed": 1376256,
    "memoryUsageAfterGc.SurvivorSpace.init": 1114112,
    "memoryUsageAfterGc.SurvivorSpace.max": 35782656,
    "memoryUsageAfterGc.SurvivorSpace.used": 547192,
    "memoryUsageAfterGc.TenuredGen.committed": 26701824,
    "memoryUsageAfterGc.TenuredGen.init": 22413312,
    "memoryUsageAfterGc.TenuredGen.max": 715849728,
    "memoryUsageAfterGc.TenuredGen.used": 18922936,
    "ingestion_time": "2019-04-02T12:17:58+00:00",
    "tag": "tml-tm.4a193af10aba.metrics.jvm.proxy.young_gc"
  }
}{
  "time": 1554207478,
  "message": {
    "memoryUsageBeforeGc.CodeCache.committed": 15925248,
    "memoryUsageBeforeGc.CodeCache.init": 2555904,
    "memoryUsageBeforeGc.CodeCache.max": 251658240,
    "memoryUsageBeforeGc.CodeCache.used": 15751744,
    "memoryUsageBeforeGc.CompressedClassSpace.committed": 5242880,
    "memoryUsageBeforeGc.CompressedClassSpace.init": 0,
    "memoryUsageBeforeGc.CompressedClassSpace.max": 1073741824,
    "memoryUsageBeforeGc.CompressedClassSpace.used": 4136776,
    "memoryUsageBeforeGc.EdenSpace.committed": 11010048,
    "memoryUsageBeforeGc.EdenSpace.init": 8912896,
    "memoryUsageBeforeGc.EdenSpace.max": 286326784,
    "memoryUsageBeforeGc.EdenSpace.used": 11010048,
    "memoryUsageBeforeGc.Metaspace.committed": 41418752,
    "memoryUsageBeforeGc.Metaspace.init": 0,
    "memoryUsageBeforeGc.Metaspace.max": -1,
    "memoryUsageBeforeGc.Metaspace.used": 37438280,
    "memoryUsageBeforeGc.SurvivorSpace.committed": 1376256,
    "memoryUsageBeforeGc.SurvivorSpace.init": 1114112,
    "memoryUsageBeforeGc.SurvivorSpace.max": 35782656,
    "memoryUsageBeforeGc.SurvivorSpace.used": 523144,
    "memoryUsageBeforeGc.TenuredGen.committed": 26701824,
    "memoryUsageBeforeGc.TenuredGen.init": 22413312,
    "memoryUsageBeforeGc.TenuredGen.max": 715849728,
    "memoryUsageBeforeGc.TenuredGen.used": 18922936,
    "ingestion_time": "2019-04-02T12:17:58+00:00",
    "tag": "tml-tm.4a193af10aba.metrics.jvm.proxy.young_gc"
  }
}{
  "time": 1554207478,
  "message": {
    "collectionCount": 19,
    "collectionTime": 83,
    "name": "Copy",
    "gcStartTime": 2950514,
    "gcEndTime": 2950518,
    "gcDuration": 4,
    "ingestion_time": "2019-04-02T12:17:58+00:00",
    "tag": "tml-tm.4a193af10aba.metrics.jvm.proxy.young_gc"
  }
}
Diagnostic Recipes / Alerts

Oracle's Garbage Collection Tuning guide: https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/index.html 

{
  "time": 1555422289,
  "message": {
    "tag": "tml-tm.tm-deploy-0-5d6f55b566-zdplp.metrics.jvm.proxy.threading",
    "ingestion_time": "2019-04-16T13:44:49+00:00",
    "TML_CLUSTER_NAME": "Tibco Mashery Local Reference Cluster Pune",
    "TML_ZONE_NAME": "us-east-1e",
    "0.AllThreadIds.0": 151,
    "0.AllThreadIds.1": 150,
    "0.AllThreadIds.2": 149,
    "0.AllThreadIds.3": 105,
    "0.AllThreadIds.4": 104,
    "0.AllThreadIds.5": 100,
    "0.AllThreadIds.6": 99,
    "0.AllThreadIds.7": 98,
    "0.AllThreadIds.8": 97,
    "0.AllThreadIds.9": 96,
    "0.AllThreadIds.10": 95,
    "0.AllThreadIds.11": 94,
    "0.AllThreadIds.12": 93,
    "0.AllThreadIds.13": 92,
    "0.AllThreadIds.14": 91,
    "0.AllThreadIds.15": 90,
    "0.AllThreadIds.16": 89,
    "0.AllThreadIds.17": 88,
    "0.AllThreadIds.18": 87,
    "0.AllThreadIds.19": 86,
    "0.AllThreadIds.20": 85,
    "0.AllThreadIds.21": 84,
    "0.AllThreadIds.22": 83,
    "0.AllThreadIds.23": 82,
    "0.AllThreadIds.24": 81,
    "0.AllThreadIds.25": 80,
    "0.AllThreadIds.26": 79,
    "0.AllThreadIds.27": 78,
    "0.AllThreadIds.28": 77,
    "0.AllThreadIds.29": 76,
    "0.AllThreadIds.30": 75,
    "0.AllThreadIds.31": 74,
    "0.AllThreadIds.32": 73,
    "0.AllThreadIds.33": 72,
    "0.AllThreadIds.34": 71,
    "0.AllThreadIds.35": 70,
    "0.AllThreadIds.36": 69,
    "0.AllThreadIds.37": 68,
    "0.AllThreadIds.38": 67,
    "0.AllThreadIds.39": 66,
    "0.AllThreadIds.40": 65,
    "0.AllThreadIds.41": 64,
    "0.AllThreadIds.42": 63,
    "0.AllThreadIds.43": 62,
    "0.AllThreadIds.44": 61,
    "0.AllThreadIds.45": 60,
    "0.AllThreadIds.46": 59,
    "0.AllThreadIds.47": 58,
    "0.AllThreadIds.48": 57,
    "0.AllThreadIds.49": 56,
    "0.AllThreadIds.50": 55,
    "0.AllThreadIds.51": 54,
    "0.AllThreadIds.52": 53,
    "0.AllThreadIds.53": 52,
    "0.AllThreadIds.54": 51,
    "0.AllThreadIds.55": 50,
    "0.AllThreadIds.56": 49,
    "0.AllThreadIds.57": 48,
    "0.AllThreadIds.58": 47,
    "0.AllThreadIds.59": 46,
    "0.AllThreadIds.60": 45,
    "0.AllThreadIds.61": 44,
    "0.AllThreadIds.62": 43,
    "0.AllThreadIds.63": 42,
    "0.AllThreadIds.64": 41,
    "0.AllThreadIds.65": 40,
    "0.AllThreadIds.66": 39,
    "0.AllThreadIds.67": 38,
    "0.AllThreadIds.68": 37,
    "0.AllThreadIds.69": 36,
    "0.AllThreadIds.70": 35,
    "0.AllThreadIds.71": 34,
    "0.AllThreadIds.72": 33,
    "0.AllThreadIds.73": 32,
    "0.AllThreadIds.74": 31,
    "0.AllThreadIds.75": 30,
    "0.AllThreadIds.76": 29,
    "0.AllThreadIds.77": 28,
    "0.AllThreadIds.78": 27,
    "0.AllThreadIds.79": 26,
    "0.AllThreadIds.80": 25,
    "0.AllThreadIds.81": 24,
    "0.AllThreadIds.82": 21,
    "0.AllThreadIds.83": 20,
    "0.AllThreadIds.84": 19,
    "0.AllThreadIds.85": 17,
    "0.AllThreadIds.86": 15,
    "0.AllThreadIds.87": 12,
    "0.AllThreadIds.88": 11,
    "0.AllThreadIds.89": 10,
    "0.AllThreadIds.90": 4,
    "0.AllThreadIds.91": 3,
    "0.AllThreadIds.92": 2,
    "0.CurrentThreadCpuTime": 2345099757,
    "0.CurrentThreadCpuTimeSupported": "true",
    "0.CurrentThreadUserTime": 1760000000,
    "0.DaemonThreadCount": 17,
    "0.ObjectMonitorUsageSupported": "true",
    "0.ObjectName.canonicalKeyPropertyListString": "type=Threading",
    "0.ObjectName.canonicalName": "java.lang:type=Threading",
    "0.ObjectName.domain": "java.lang",
    "0.ObjectName.domainPattern": "false",
    "0.ObjectName.keyPropertyList.type": "Threading",
    "0.ObjectName.keyPropertyListString": "type=Threading",
    "0.ObjectName.pattern": "false",
    "0.ObjectName.propertyListPattern": "false",
    "0.ObjectName.propertyPattern": "false",
    "0.ObjectName.propertyValuePattern": "false",
    "0.PeakThreadCount": 95,
    "0.SynchronizerUsageSupported": "true",
    "0.ThreadAllocatedMemoryEnabled": "true",
    "0.ThreadAllocatedMemorySupported": "true",
    "0.ThreadContentionMonitoringEnabled": "false",
    "0.ThreadContentionMonitoringSupported": "true",
    "0.ThreadCount": 93,
    "0.ThreadCpuTimeEnabled": "true",
    "0.ThreadCpuTimeSupported": "true",
    "0.TotalStartedThreadCount": 144
  }
}
Metric Field Name / Computation Unit Data Type Details
Max Used Threads 0.PeakThreadCount NA Number Represents peak live thread count since the JVM started or the peak was reset.
Current Thread Count 0.ThreadCount NA Number Represents current number of live daemon and non-daemon threads.
Started Thread Count 0.TotalStartedThreadCount NA Number Total number of threads created and/or started since the JVM started.