Sizing Metrics

Shared memory usage has these parts:

The base system usage for a Spotfire Streaming node can be determined with epadmin commands like the following, directed at an unloaded node (that is, with no application running, no application usage of memory):

epadmin [--ad=adminport|--servicename=cluster.node] enable statistics –stat=memoryusage
epadmin [--ad=adminport|--servicename=cluster.node] display statistics –stat=memoryusage

As shown, a Spotfire Streaming node base system usage is 21% of 512 megabytes, or 100 megabytes. This base footprint will increase slightly as managed object classes are loaded.

Managed objects persist their state in shared memory until they are explicitly deleted by the application. The shared memory usage of a managed object can be determined programmatically by creating an instance of the managed object, populating it with typical data values, and passing it to the com.kabira.platform.osstats.Type.memorySize()method.

This is shown in Example 1, “Object size snippet”:

package com.tibco.ep.dtm.snippets.sizing;

import com.kabira.platform.ManagedObject;
import com.kabira.platform.Transaction;
import com.kabira.platform.annotation.Key;
import com.kabira.platform.annotation.Managed;
import com.kabira.platform.osstats.Type;

/**
 *  Display Managed object memory usage.
 */
public class ObjectSize
{
    /**
     * Sample application object
     */
    @Managed
    @Key(name = "ByName", fields =
    {
        "name"
    }, unique = true, ordered = false)
    private static class MyObject
    {
        static final int NUMBER_OF_ELEMENTS = 100;
        String[] stringArray;
        @SuppressWarnings("unused")
        final String name;

        /**
         * When created, populate this instance with some data
         * @param name Name value
         */
        public MyObject(String name)
        {
            this.name = name;

            stringArray = new String[NUMBER_OF_ELEMENTS];

            for (int i = 0; i < NUMBER_OF_ELEMENTS; i++)
            {
                stringArray[i] = Integer.toString(i);
            }
        }
    }

    /**
     * Main entry point
     * @param args  Not used
     */
    public static void main(String[] args)
    {
        //
        //  Create the objecct
        //
        new Transaction("Create Object")
        {
            @Override
            protected void run() throws Rollback
            {
                new MyObject("Sample");
            }
        }.execute();

        //
        //  Report object size - this must be done in a separate 
        //  transaction.  It only works for committed objects.
        //
        new Transaction("Report Object Size")
        {
            @Override
            protected void run() throws Transaction.Rollback
            {
                for (MyObject myObject : ManagedObject.extent(MyObject.class))
                {
                    System.out.println(new Type().memoryUsage(myObject));
                    ManagedObject.delete(myObject);
                }
            }
        }.execute();
    }
}

This program's output is similar to the following:

Allocation type: # of bytes, allocator bucket size, notes
=====================================================================
metadata: 480, 592, spaces: [allocation=64] [system=24] [lock=112]
key com.kabira.snippets.sizing.ObjectSize$MyObject::ByName: 128, 208
array com.kabira.snippets.sizing.ObjectSize$MyObject::stringArray: 1006 (aligned 1008), 1168
optimal allocationSpaceBytes = 1032
event queue: 0, 0
Total: 1614 1968

The output has three columns:

The final line of the report shows the total memory requested and the actual memory allocated.

An addition line of the report shows the optimal size for the allocation space. At object creation, by default, the Runtime chooses a small value for the allocation space. User data for strings and arrays that does not fit within this space causes additional memory allocations. If the optimal size is different than the allocation space shown in the metadata line of the report space and performance savings may be gained by explicitly setting it via the allocationSpaceBytes element of the Managed annotation. From the example above, change the setting to 1032 bytes:

      @Managed(allocationSpaceBytes = 1032)
      @Key(name = "ByName", fields =
      {
         "name"
      }, unique = true, ordered = false)
      private static class MyObject      

Re-running the modified class, results in:

      Allocation type: # of bytes, allocator bucket size, notes
      =====================================================================
      metadata: 1336, 1424, spaces: [allocation=1032] [system=24]
      key com.kabira.snippets.sizing.ObjectSize$MyObject::ByName: 136, 240
      optimal allocationSpaceBytes = 1032
      event queue: 0, 0
      Total: 1472 1664

Transaction allocation pages are created during in-flight transactions to manage write and read images (snapshots) of the object. The memory used for in-flight transactions pages is equivalent to the size of the object fields. The transaction page is released when the transaction commits or aborts. The total amount of memory consumed for transaction pages should be multiplied by the number of concurrent transactions to get the total impact on shared memory size. For example, if a system is running at 1000 transactions per second, and each transaction creates a transaction page of a 50 byte object, the in-flight transaction log size is 1000 * 50, or 50,000 bytes.

Finally, any replica objects on a node consume shared memory. The amount of shared memory consumed by a replica object is the same as the shared memory consumed by the object on its active node, which can be calculated using the com.kabira.platform.Type.memoryUsage()method as discussed above.

JVM heap memory usage in Spotfire Streaming follows normal JVM heap usage, with the following differences:

A Spotfire Streaming node consists of a small number of processes communicating through shared memory to provide the Spotfire Streaming Runtime services. The total size of the code that may be executed by these processes is approximately 100 megabytes. This is a system wide (per-server) cost, and not a per-process cost, because the code is contained in shared object files.

A typical UNIX installation requires adding at least as much swap space as there is physical memory. However it is highly recommended that a machine running a Spotfire Streaming system have enough physical memory so that it never needs to swap. Swapping runs at disk access speeds, while Spotfire Streaming is designed to run from memory at memory access speeds.