Troubleshooting

High CPU Usage in NoSQL Container

It has been observed that nodes where NoSQL container run are showing high CPU usage on some Linux platforms, despite limited use of Cassandra inside NoSQL per container. Although there are various known reasons of high CPU usage by Cassandra, one possible general cause of high CPU usage on Linux platform is Transparent Hugepages. Many Linux distributions come with Transparent Hugepages enabled by default. When Linux uses Transparent Hugepages, the kernel tries to allocate memory in large chunks (usually 2 MB), rather than 4K. However, some applications such as Cassandra, still allocate memory based on 4K pages. This can cause noticeable performance problems when Linux tries to defrag 2 MB pages.

To confirm this behavior and to reduce the CPU usage, follow the steps below. If the reason is confirmed as described below, you should stop the Mashery Local cluster, if possible, as it requires restarting the nodes.

1. SSH in the node.

   cat /sys/kernel/mm/transparent_hugepage/enabled

   You should get the following output:

   [always] madvise never

2. If you see this output, open the file /etc/rc.local for editing. It will have only one statement (exit 0):

   sudo vi /etc/rc.local

3. Add the following lines before the last statement (exit 0):

   if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   
   echo never > /sys/kernel/mm/transparent_hugepage/enabled
   
   fi
   
   if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
   
   fi

   The file should have the following content after the modification.
   
   
4. Save and exit.
5. Restart the node. Please perform steps 1-5 on all the nodes.
6. After successful restart, SSH into the node and run the command from step 1.
   You should get the following output:

   [always] madvise never

7. Once all nodes are restarted, you can redeploy the Mashery Local cluster.