Characteristics of In-Memory Tables

The TIBCO Patterns in-memory table is not a persistent data store; it must be reloaded each time the matching server is restarted. TIBCO Patterns in-memory tables are designed to work with, not to replace, your current DBMS or another persistent data store.

TIBCO Patterns in-memory tables are quite similar to DBMS tables. An in-memory table consists of fielded records, with every record having the same number of named fields. The number of fields and their types are established at the time the table is created, and remain fixed for the lifetime of the table – addition or deletion of fields to an in-memory Patterns table is not supported.

Each record in the table must have a unique key value. Keys can be defined by the application or auto-generated by the TIBCO Patterns server.

A TIBCO Patterns in-memory table has a fixed configuration that is defined at the time of creation. It cannot be changed for the life of the table. The configuration includes the list of fields and the field types, character maps associated with each field, and indexes for the table including the index (prefilter) used for inexact matching and any partition indexes used for predicate tests. It also includes any parent-child relationship between tables.

Field Types

The set of possible field types includes: searchable text, Variable Attributes, non-searchable text, integer, float, date, and date-time.

Searchable Text

Searchable text is the default field type, and by far the most commonly used. Searchable text consists of UTF-8 encoded textual data that is used for inexact matching. Such text can also be used for exact matching within predicate expressions. See Predicates.

Variable Attributes

A field of type Variable Attributes is a container for an arbitrary set of name-value pairs. The value, like searchable text fields, is any UTF-8 encoded text value. The name is treated much as a field name. Each Variable Attribute name-value pair can be referenced by inexact matching queries and exact matching predicates like a standard searchable text field. Unlike standard fields, however, the names of attributes are not predefined and each record can contain a completely different set of attributes.

A table can contain at the most one field of Variable Attributes type. It is an optional field and if present, it is always the last field in the table. This field is named like any other field in the table.

Methods are provided for adding Variable Attributes to records. Methods are also provided for encoding a set of Variable Attributes into a text string that can be included in a standard CSV file. This allows tables containing Variable Attributes to be dumped to and loaded from a CSV file.

A convention for referencing a Variable Attribute value within a table is: 

   <field-name>:<attribute-name>

where:

— field-name is the name of a field of type Variable Attributes and
— attribute-name is the name of a Variable Attribute.

For example, if you define the "address" field of type Variable Attributes, it can be defined using many attributes which could be queried using expressions such as: "address:line1", "address:line2", "address:city".

The predicate expression from the example in the Introduction can be: $"address:state" = "CA"

Note: This convention imposes the restriction that field names cannot contain the ‘:’ (colon) character. This restriction is enforced for all fields.

Although limits are present on the number and size of Variable Attributes within a record and table, you are unlikely to encounter them in common scenarios. For more information, see the "Variable Attributes - Usage Details" section in TIBCO Patterns Programmer’s Guide.

Non-searchable Text

Non-searchable text is UTF-8 encoded textual data that cannot be used for inexact matching. However, you can store it in the in-memory table to avoid extra calls to the permanent data store, or use it for exact matching in predicate expressions, or for some other purpose. When inexact matching is not required for textual data, the use of this field type can significantly reduce memory usage and load times.

Integer and Float

Integer and float are integers or floating-point values that are transmitted to the TIBCO Patterns server as text, but stored in the table as binary values. Fields of these types are primarily useful in predicate expressions, in which they are more efficient, and require less storage, than the equivalent text value. Two special values recognized for these field types are:

— empty, which indicates no data in the field;
— invalid, which indicates that the data transmitted for the field could not be successfully translated to the integer or float type. For example, a string value of “apple” is unable to be translated to an integer type, resulting in an invalid value of the field for that record.

Date and Date-time

The date and date-time fields store date and date-time values compactly. You cannot use these fields in the standard inexact queries. However, a specialized inexact matching query for date fields (but not date-time fields) provides better results than matching on the text representation of the date value. You can use both field types in predicate expressions.

A standard date field is not visible to the GIP prefilter (For information about the GIP prefilter, see Prefilters and Scaling). This can result in a loss of accuracy as the GIP prefilter cannot use the date field information to help select the best matching records. However, a date field can be marked as a searchable date field. A searchable date field is visible to the GIP prefilter. This can improve search accuracy, but increases memory usage. A searchable date field is still a date field, it cannot be used in a standard inexact text matching query.

Memory Utilization

It is a good practice to load only the fields that are required for matching in in-memory Patterns tables. If the table contains additional fields (for example, to avoid extra calls to retrieve records from a DBMS), use the non-searchable text type for these fields to minimize memory usage.

As a general rule of thumb, an in-memory Patterns table consisting entirely of searchable text fields consumes approximately five times the amount of storage represented by those fields. This is because of the specialized indices that are built at load time to support high-performance inexact matching. Non-searchable fields (text, integer, float, date, or date-time) do not utilize any extra memory.