Debugging High Latency Issues

The latency of particular service depends on the complexity of service implementation, payload, workload, the number of services deployed on a container, and CPU or memory resources made available to the container.

Prerequisites

Assuming all the components of the engine are tuned for debugging high latency issues on the container, collect the following data that helps in further debugging and understanding the issues.

Procedure

  1. Capture the process execution statistics for the test run. This would help analyze the time spent in individual processes and activities. For more information, see TIBCO BusinessWorks™ Container Edition Application Monitoring and Troubleshooting guide.
  2. Capture thread dumps for analyzing the thread state and calls. Capture 5 thread dumps at an interval of 5 seconds each. Redirect all the collected threads dumps to separate files.
    • The thread dumps can be captured using jstack utility shipped with JDK. 
      ./jstack <PID of container application> > ThreadDump_n.txt
    • The thread dumps can be captured through JConsole or JVisualVM utilities.
  3. Capture top CPU consuming threads data for 5 minutes run by using jvmtop utility.
    For more information about jvmtop utility, see support article KB000034702. 
    ./jvmtop <PID of container application> > JVM_topthreads.txt
  4. Capture method level CPU profiling data for 5 minutes run by using the jvmtop utility. 
    ./jvmtop --profile <PID of container application> > JVM_CPUProfile.txt
  5. Capture container application logs for the test run duration.
  6. Capture the CPU and the memory utilization data for 5 runs on the container application.
    • The data can be captured through the top utility on Unix. 
      top -p <PID of container application> > top_application.txt
    • The data can also be captured through JConsole or JVisualVM utilities.
  7. Check the BW_JAVA_OPTS and other run-time parameters with which the container is running. It helps analyze the JVM parameters and other engine tuning parameters.
  8. Capture system configurations of servers such as CPU details, RAM, and number of cores where TIBCO BusinessWorks Container Edition container application, external services, and load generator are running. Capture details of /proc/meminfo and /proc/cpuinfo files. 
    cat /proc/meminfo and cat /proc/cpuinfo