Debugging High Memory Utilization Issues

The memory utilization for a particular service on the AppNode depends on the complexity of service implementation, payload, workload, the number of services deployed on the AppNode, CPU, or memory resources made available to the AppNode.

Before you begin

Assuming all the components of the engine are tuned for debugging high memory utilization issues on the AppNode, collect the following data that helps in further debugging and understanding the issues.

    Procedure
  1. Capture the heap dump on the AppNode when memory issues are observed on the AppNode.
    The heap dumps can be captured by using the jmap utility. 
    jmap -dump:live,file=memorydump.hprof <PID of appnode>
    The heap dumps can also be captured through JConsole or JVisualVM utilities.

    The heap dump can be analyzed using the memory analyzer tool for checking memory leaks and top components of memory.

  2. Capture jstat data for checking the allocation and utilization of different memory pools for 5-minutes run. 
    jstat –gc <PID of appnode> > jstat_gc.txt
  3. Capture thread dumps for analyzing the thread state and calls. Capture five thread dumps at an interval of 5 seconds each. Redirect all the collected threads dumps to separate files.
    The thread dumps can be captured by using the jstack utility shipped with JDK. 
    ./jstack <PID of appnode> > ThreadDump_n.txt
    The thread dumps can also be captured through JConsole or JVisualVM.
  4. Capture AppNode logs for the test run duration.
  5. Capture the CPU and memory utilization data of the AppNode for 5-minutes run.
    The data can be captured through the top utility on Unix. 
    top –p <PID of > > top_appnode.txt
    The data can also be captured through JConsole or JVisualVM utilities.
  6. Capture the AppNode config.ini and TRA file of the AppNode. This helps analyze the JVM parameters and other engine tuning parameters.
  7. Capture system configurations of servers such as CPU details, RAM, and number of cores where ActiveMatrix BusinessWorksAppNode, external services, and load generator are running. Capture details of /proc/meminfo and /proc/cpuinfo files. 
    cat /proc/meminfo and cat /proc/cpuinfo