Data Persistence Directories

To use the Checkpoint/Restore feature, or the Durable Data feature, a separate directory must be allocated for each running TIBCO Patterns server. This directory is called the Data Persistence directory. It is also sometimes referred to as the Checkpoint/Restore directory, although this name is obsolete. The server must have permission to add and delete files in this directory. The Checkpoint/Restore and Durable Data features store data in this directory. No two running TIBCO Patterns servers can share the same data-persistence directory. Also, no other process must attempt to modify this directory or its contents in any way. When using the directory for the first time, it must be empty. After this, the directory is managed by the TIBCO Patterns server.

In a typical production environment, there would be one TIBCO Patterns server per machine. A data-persistence directory is created on the machine. The TIBCO Patterns server must always run as the same user that owns the data-persistence directory. Typically, a special user ID is created for the TIBCO Patterns server and the user owns the data-persistence directory. The server is then started using the -R directory argument, where directory is the full path name to the created directory. The TIBCO Patterns server then takes over the responsibility for managing the contents of the directory, and no user action is needed. When a server starts, it performs the necessary cleanup of incomplete operations.

Selecting a Data-Persistence Directory

Consider the following when allocating a data-persistence or restore directory:

•

Accessibility. The TIBCO Patterns servers must be able to access the directory from the home directory where the server was started.

•

File Name Limits. The file system must support long file names.

•

Space. The file system or partition on which the directory resides must have enough space to store the data. For the checkpoint/restore feature, files may take up to twice the size of the raw data. For the Durable-data feature, an additional 5x space is needed for ongoing record updates. So, if a CSV file dump of the table data takes about 1GB, the checkpoint file may take up to 2GB, and durable-data updates take up to 6GB. Ensure that enough space is available on the device as TIBCO Patterns servers do not check for the availability of space before writing out the data.

•

File Size Restrictions. The checkpoint/restore feature stores the data for each table in a single file. Durable data splits the data for each table across several files, but it stores approximately three times as much of the total data. For large tables, the files created by either feature could exceed the maximum file size on some file systems. Ensure that the file system supports sufficiently large files to accommodate your tables.

•

Network Traffic. If the directory is on a remote mounted file system (for example, NFS), reads and writes to the data-persistence directory could generate a very large amount of network traffic which could affect overall network performance.

•

Speed. Directories on remote devices or slow devices can significantly impact checkpoint/restore and durable-data performance. A fast local device is recommended.

•

Reliability. Do not depend on the TIBCO Patterns servers, even when using checkpoints or durable-data, as your primary secure data repository. However, if losing the data-persistence directory significantly impacts the availability of critical services, consider putting the data-persistence directory on a mirrored or RAID device.

•

Each running TIBCO Patterns server must have its unique data-persistence directory.

Restoring Checkpoint Data at Server Start-up

Specifying the -A (--auto-restore) option in addition to the -R (--restore-dir) option instructs the TIBCO Patterns to restore checkpointed data at start-up. All checkpointed tables, thesauri, models, and character-maps are restored. If any checkpointed object fails to restore, the server logs an error and exits.

Sometimes, when running multiple TIBCO Patterns servers containing the same tables, it might be convenient to keep a single copy of the tables and have all TIBCO Patterns servers load from this shared copy by using the -a (--restore-from) option. It loads all the checkpointed tables from the named directory. This does not enable checkpointing in the shared directory; it is a read-only operation. The shared directory and all of its files must be readable by the TIBCO Patterns server. The shared directory must not contain incomplete checkpoints. The TIBCO Patterns server validates the contents of the directory, and exits if an incomplete checkpoint is found in the shared directory.

Checkpointing, auto-restore, and shared restore might be combined. Data in the shared directory is loaded first, followed by data from the data-persistence directory. This might override the shared data.

To manage multiple TIBCO Patterns servers with a common data set, designate one TIBCO Patterns server as the master. This server is always started first and has checkpoint and restore enabled on the directory (use the -R and -A options). When started, this master server cleans up any incomplete operations left in the directory.

Data-Persistence Directory Contents

The contents of the directory are managed by the TIBCO Patterns server. The files in the directory are briefly described in the following table:

File name
LockFlag	Used to coordinate access to the directory.
object-name-type-flagO000	A checkpoint file for the object named object-name. This is a binary data file. It contains the complete information for the in-memory object needed to restore the object. The type-flag is a one character flag indicating the object type. The possible values for the type-flag are: D : a Data table T : a Thesaurus table C : a Character map M : a Learn Model An example for the Learn model is Model1.MO000
object-name-type-flagOhhh	A checkpoint file for a previous version of the object named object-name. The type-flag is as described in the previous entry. This exists only if the system fails during a checkpoint operation. The TIBCO Patterns server cleans this up on start up. This is the original version of the checkpoint file that is being modified by the open transaction with the short ID: hhh, where hhh is three hex digits. A transaction never has a short ID of '000' and therefore, this is always distinct from the current committed version that has the suffix type-flagO000. This file is deleted when the associated transaction is closed.
object-name-type-flagNhhh	A new, possibly incomplete, checkpoint file. The file was created by the open transaction with the short ID: hhh, where hhh is three hex digits. The file is removed when the transaction is closed.
object-name-type-flagRhhh	This indicates that the object with the name: object-name and the type indicated by the type-flag was renamed by the open transaction with the short ID: hhh, where hhh is three hex digits. This file is deleted when the associated transaction is closed.
hhh.ipt	A log of checkpoint restore actions performed by the transaction with the short ID: hhh, where hhh is three hex digits. This file is deleted when the associated transaction is closed.
OINT ODEF OCONT	Sub-directories used by the Durable-Data feature.
OINT/hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh.ointOINT/hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh.tmp	Contains naming and object-level transactional information about a data object.
ODEF/hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh.defODEF/hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh.tmp	Contains the immutable content of a data object (for example, the field names of a table).
OCONT/hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh.*	Contains mutable data of a data object (for example, table records).