Jemalloc and Jeprof

Jemalloc tool tracks down native memory allocations and identifies native memory leak suspects.

1. Download jemalloc from GitHub.com.
2. Install the tool by following the steps specified at GitHub.com.
3. Once jemalloc is build include jemalloc in BWCE buildpack.
4. Edit the bwce-buildpack/resources/prestart.sh file and add the following commands before the exec ./tibco.home/bw*/*/bin/startBWAppNode.sh command:

   export LD_PRELOAD=/root/jemalloc/jemalloc-stable-4/lib/libjemalloc.so

   export MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17,prof_final:true,prof_leak:true

5. When the server is started and the memory allocation is done, jeprof*.heap files are generated at ./tibco.home/bw*/*/bin/ folder.
   As the memory utilization grows, more files would be generated.
6. Analyze the heap files with the jeprof command. JDK needs to be installed or included in the buildpack for running jeprof commands as it uses local java for analyzing the jeprof*.heap files.

   jeprof --show_bytes <PATH to java> jeprof.*.heap

   The jeprof utility is included in the jemalloc\bin folder. After exection of jeprof command, jeprof console opens.

7. Type top when you reach the jeprof console. For example:

   jeprof --show_bytes /usr/lib/jvm/java-8-oracle/jre/bin/java jeprof*.heap
   Using local file /usr/bin/w.
   Using local file jeprof.19678.0.f.heap.
   Welcome to jeprof! For help, type 'help'.
   (jeprof) top
   

   It shows the following output:

    Total: 267184 B
   258032 96.6% 96.6% 258032 96.6% _3_2_5
   3616 1.4% 97.9% 3616 1.4% _nl_intern_locale_data
   2048 0.8% 98.7% 2208 0.8% __tzfile_read
   1024 0.4% 99.1% 1024 0.4% getpwnam
   1024 0.4% 99.5% 1072 0.4% getpwuid 
   448 0.2% 99.6% 448 0.2% __gconv_lookup_cache
   224 0.1% 99.9% 224 0.1% strdup 
   160 0.1% 99.9% 160 0.1% __tzstring 
   128 0.0% 100.0% 3760 1.4% _nl_load_locale_from_archive 
   48 0.0% 100.0% 48 0.0% get_mapping

8. To run the jeprof command on a single file, use the following command:

   jeprof --show_bytes <PATH to java> <Heap file name>

   After exection of jeprof command, jeprof console opens.

9. Type top when you reach the jeprof console. For example:

   jeprof --show_bytes /usr/lib/jvm/java-8-oracle/jre/bin/java jeprof.19678.0.f.heap
    Using local file /usr/bin/w.
    Using local file jeprof.19678.0.f.heap.
   Welcome to jeprof! For help, type 'help'.
    (jeprof) top
   			 

   It shows the following output:

   Total: 267184 B
   258032 96.6% 96.6% 258032 96.6% _3_2_5
   3616 1.4% 97.9% 3616 1.4% _nl_intern_locale_data
   2048 0.8% 98.7% 2208 0.8% __tzfile_read
   1024 0.4% 99.1% 1024 0.4% getpwnam
   1024 0.4% 99.5% 1072 0.4% getpwuid 
   448 0.2% 99.6% 448 0.2% __gconv_lookup_cache
   224 0.1% 99.9% 224 0.1% strdup 
   160 0.1% 99.9% 160 0.1% __tzstring 
   128 0.0% 100.0% 3760 1.4% _nl_load_locale_from_archive 
   48 0.0% 100.0% 48 0.0% get_mapping

10. To stop profiling once the analysis is done and leak suspects are identified run command:

    unset MALLOC_CONF

    If profiling is not stopped, the jeprof heap files are continuously generated.