Messages : Store Messages in Multiple Stores

Store Messages in Multiple Stores
As described in Message Delivery Modes, the EMS server writes PERSISTENT messages to disk while waiting for confirmation of receipt from the subscriber. Messages are persisted to a store. The EMS server can write messages to different types of stores: file-based stores, mstores, and database stores.
By default, the EMS server writes persistent messages to file-based stores. There are three default store files, as described in Default Store Files. You can configure the system to change the default store files and locations, and also to store persistent messages to one or more store files, filtering them by destination. Stores are defined in the stores.conf configuration file, and associated with a destination using the store destination property.
Stores have properties that allow you to control how the server manages the store files. For example:
With the multiple stores feature, you can configure your messaging application to store messages in different locations for each application, or to create separate files for related destinations. For example, you can create one store for messages supporting Marketing, and one for messages supporting Sales. Because stores are configured in the server, they are transparent to clients.
The EMS Administration Tool allows administrators to review the system’s configured stores and their settings by using the show stores and show store commands.
Store Types
TIBCO Enterprise Message Service allows you to configure several different types of stores, described here.
File-Based Stores
The EMS server stores persistent messages in file-based stores. You can use the default store files, or create your own file-based stores. You direct the EMS server to write messages to these store files by associating a destination with a store.
File-based stores are enabled by default, and the server automatically defines three default stores, described below. You do not need to do anything in order to use the default stores.
The section Configuring Multiple Stores describes how to change store settings or create custom stores.
mstores
The mstore is designed to recover quickly after a failover. When mstores are in use, the EMS server starts quickly, but may run more slowly until the mstore cache is fully loaded. This is because the EMS server continually monitors the store in the background. The server reads through the mstore incrementally and discards stale data, such as purged and expired messages.
As a result, expired and purged messages are not immediately removed from the mstore, and may remain in the store longer than they would in a file-based or database store—although they are not delivered to the consumer. These messages are discarded during the periodic scans of the store. The scanning behavior is determined by parameter settings in the store configuration, and is further described in Understanding mstore Intervals.
Because of this behavior, querying the server for a total pending message count may return an inaccurate value. However, querying specific destinations returns an accurate count.
The section Configuring Multiple Stores describes the mstore configuration process.
Database Stores
The EMS server can store messages in one or more database instances. Database stores must be configured to use a supported database. See Chapter 10, Using Database Stores for a full description of this feature.
Default Store Files
The EMS server defines these default store files, and writes persistent messages and meta data to them:
$sys.nonfailsafe—Persistent messages without a store property designation are written to $sys.nonfailsafe by default. The server writes messages to this store using asynchronous I/O calls.
$sys.failsafe—Associate a destination with this store to write messages synchronously. The server writes messages to this store using synchronous I/O calls.
$sys.meta—The server writes state information about durable subscribers, fault-tolerant connections, and other metadata in this store.
The EMS server creates these file-based stores automatically, and no steps are required to enable or deploy them. However, you can change the system configuration to customize the default store file settings, or even override the default store settings to either point to different file location, or write to an mstore or database.
Note that the $sys.meta store may not be reconfigured to use the mstore type.
Configuring Multiple Stores
This section describes the basic steps required to configure file-based stores and mstores. Database store configuration is detailed in Chapter 10, Using Database Stores.
Settings for creating and configuring multiple stores are managed in the EMS server, and are transparent to clients. To configure the multiple stores feature, follow these steps:
1.
Stores are created and configured in the stores.conf file. Each store must have a unique name. The stores are configured through parameters.
File-based stores have two required parameters, type and file, which determine that the store is a file-based store, and set its location and filename. Optional parameters allow you to determine other settings, including how messages are written to the file, the minimum size of the file, and whether the EMS server attempts to truncate the file.
mstores also have two required parameters, type and file. Optional parameters configure the scan interval, during which expired and purged messages are removed. See Understanding mstore Intervals below for information about interval settings.
2.
Messages are sent to different stores according to their destinations. Destinations are associated with specific stores with the store parameter in the topics.conf and queues.conf files.
File-Based Stores
When using file-based stores, you can also change store associations dynamically using the setprop topic or setprop queue command in the EMS Administration Tool.
mstores
When using mstores, you cannot dynamically change the mstore associations after they have been set. In order to change a destination’s store property from a store of the type mstore:
a.
b.
c.
d.
 
Once mstores are enabled for a destination, you cannot dynamically change the store property value using setprop or addprop. To change the store property, you must stop the server, empty the mstore, manually make the change, and restart.
The mstore stores data in multiple files. As a result, mstores cannot operate in out of space conditions. In order to prevent an out of space situation from arising, we recommend ensuring that there is at least twice as much disk space available for the mstore as needed to hold the maximum amount of data that might be stored in it.
Multiple destinations can be mapped to the same store, either explicitly or using wildcards. Even if no stores are configured, the server sends persistent messages that are not associated with a store to default stores. See Default Store Files for more information.
For details about the store parameter, see store on page 70.
Understanding mstore Intervals
The mstore is designed to ensuring a quick EMS server start-up time. To enable this functionality, the EMS server must continually monitor the store in the background. The server reads through the mstore incrementally and discards stale data, such as purged and expired messages.
In order to keep the background activity from degrading server performance, the examination is performed in increments. The length of these increments and the amount of data processed each increment are controlled by two parameter settings. These stores.conf parameters can be configured for each mstore.
The default parameter settings are optimized for best performance in most production environments (see mstore Parameters for information about the default values). However, if the amount of data in the mstore grows significantly, the read rates associated with the background activity may begin to affect message transmission rates in the EMS server. If the EMS server performance is negatively affected by the size of the mstore, you can tune the mstore parameter values to spread mstore background activity over a longer period of time, thereby decreasing the associated read rates.
scan_target_interval — the maximum amount of time allowed before each message in the store is examined.
For example, if the scan_target_interval is 24 hours, each section of the mstore will be examined at least once every day. Because purged and expired messages are not removed from the mstore until they are examined by this background process, this means that it can take up to 24 hours before a message is removed from the queue following a purge command (making underlying storage space available for re-use).
scan_iter_interval — the length of time between each increment of background activity.
For example, if the scan_iter_interval is 10 seconds, the EMS server begins examining a new section of the mstore every 10 seconds. The amount of data read in each increment is dependent on the total size of the store and the length of the scan_target_interval. The server must examine enough data in each interval to fully traverse the store within the target interval.
Example
For example, assume that scan_iter_interval is 10 seconds, scan_target_interval is 1 day (86,400 seconds), and the mstore contains 9 GBs of data. Every 10 seconds, the EMS server will examine about 1 MB of data. This produces an average read rate of about 100 KB/sec, which is unlikely to produce performance degradation with most modern storage mediums.
If EMS server performance does slow, you may need to increase the scan_target_interval value in order to spread the background activity over longer period of time. You can monitor the settings for problems using the show store command and checking the ratio of "Discard scan bytes" to "Discard scan interval". For best results, this ratio should be kept below 20% of the disk processing capacity for each mstore. Consider this ratio in relation to your storage medium's overall data transfer capacity, so as to make sure that the background activity does not occupy an excessive amount of the system's resources.
Implications for Statistics
The background monitoring and cleanup that occurs in the mstore also affect some key server statistics. Before the first scan has been completed, some message statistics may be underreported due to purged and expired messages that the server has not yet removed. Until the first background scan is complete for some or all mstores, the server may not have an accurate messages count.
For example, when the EMS server first starts, the "Pending Messages" and "Pending Message Size" counts reported by the info command in the administration tool can be understated, because the command only reports on messages it has scanned before the command is issued. Similarly, the "Message Count" and "Message Size" reported by the show store command may report a smaller number than actually exist in the store.
Once the first scan is complete, these counts can be considered accurate. To check the scan status on an mstore, use the show store command. The statistics returned now include a "First scan finished" field, which reports the scan status since the last EMS server start time. When the value of this field is true, the server statistics can be considered accurate.
If it is important to acquire the correct values for these statistics sooner, you will need to decrease the scan_target_interval.