Jemalloc and Jeprof

Jemalloc tool tracks down native memory allocations and identifies native memory leak suspects.

Procedure

  1. Download jemalloc from GitHub.com.
  2. Install the tool by following the steps specified at GitHub.com.
  3. Set the following properties and start the BW AppNode from the same console: 
    cd [BW_HOME]/bw/6.5/domains/<DomainName>/appnodes/<AppSpaceName>/<AppNodeName>/bin
    export LD_PRELOAD=/root/jemalloc/jemalloc-stable-4/lib/libjemalloc.so
    export MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17,prof_final:true,prof_leak:true
    ./startBWAppNode.sh
  4. Once jemalloc is build include jemalloc in BWCE buildpack.
  5. Edit the bwce-buildpack/resources/prestart.sh file and add the following commands before the exec ./tibco.home/bw*/*/bin/startBWAppNode.sh command: 
    export LD_PRELOAD=/root/jemalloc/jemalloc-stable-4/lib/libjemalloc.so
    export MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17,prof_final:true,prof_leak:true
  6. When the server is started and the memory allocation is done, jeprof*.heap files are generated in same {BW_HOME}/bw/6.5/domains/<DomainName>/appnodes/<AppSpaceName>/<AppNodeName>/bin folder.
    As the memory utilization grows, more files would be generated.
  7. Analyze the heap files with the jeprof command. 
    jeprof --show_bytes <PATH to java> jeprof.*.heap

    The jeprof utility is included in the jemalloc\bin folder. After exection of jeprof command, jeprof console opens.

  8. Type top when you reach the jeprof console. For example: 
    jeprof --show_bytes /usr/lib/jvm/java-8-oracle/jre/bin/java jeprof*.heap
Using local file /usr/bin/w.
Using local file jeprof.19678.0.f.heap.
Welcome to jeprof! For help, type 'help'.
(jeprof) top

    It shows the following output: 

     Total: 267184 B
258032 96.6% 96.6% 258032 96.6% _3_2_5
3616 1.4% 97.9% 3616 1.4% _nl_intern_locale_data
2048 0.8% 98.7% 2208 0.8% __tzfile_read
1024 0.4% 99.1% 1024 0.4% getpwnam
1024 0.4% 99.5% 1072 0.4% getpwuid 
448 0.2% 99.6% 448 0.2% __gconv_lookup_cache
224 0.1% 99.9% 224 0.1% strdup 
160 0.1% 99.9% 160 0.1% __tzstring 
128 0.0% 100.0% 3760 1.4% _nl_load_locale_from_archive 
48 0.0% 100.0% 48 0.0% get_mapping
  9. To run the jeprof command on a single file, use the following command: 
    jeprof --show_bytes <PATH to java> <Heap file name>

    After exection of jeprof command, jeprof console opens.

  10. Type top when you reach the jeprof console. For example: 
    jeprof --show_bytes /usr/lib/jvm/java-8-oracle/jre/bin/java jeprof.19678.0.f.heap
 Using local file /usr/bin/w.
 Using local file jeprof.19678.0.f.heap.
Welcome to jeprof! For help, type 'help'.
 (jeprof) top

    It shows the following output: 

    Total: 267184 B
258032 96.6% 96.6% 258032 96.6% _3_2_5
3616 1.4% 97.9% 3616 1.4% _nl_intern_locale_data
2048 0.8% 98.7% 2208 0.8% __tzfile_read
1024 0.4% 99.1% 1024 0.4% getpwnam
1024 0.4% 99.5% 1072 0.4% getpwuid 
448 0.2% 99.6% 448 0.2% __gconv_lookup_cache
224 0.1% 99.9% 224 0.1% strdup 
160 0.1% 99.9% 160 0.1% __tzstring 
128 0.0% 100.0% 3760 1.4% _nl_load_locale_from_archive 
48 0.0% 100.0% 48 0.0% get_mapping
  11. To stop profiling once the analysis is done and leak suspects are identified run command: 
    unset MALLOC_CONF

    If profiling is not stopped, the jeprof heap files are continuously generated.