Database Table Retention (LX, MX, LXvirtual, MXvirtual Only)

Use the Administration > System Settings > Database Table Retention tab to configure the length of time the appliance stores database contents before purging.

Data stored in the appliance is purged in the following scenarios:
  • On expiry: All data eventually expires and is deleted. The expiry is specified in the Duration field on the Administration > System Settings > Database Table Retention page.
  • On emergency: When the disk usage reaches a threshold value defined in LxDiskUsagePercent. See Changing the Purging Database Threshold.
The archiver and the scheduler are involved in data purging.
  • The archiver handles purging of raw and index data. A file with the oldest expiration time is deleted first. To set the retention time of raw logs and index data, see Data Retention Rules.
  • The scheduler handles emergency purging of MySQL tables. A table with the oldest data is removed, regardless of log type or table type.
The archiver watches the scheduler's progress and the age of the oldest table remaining. It makes sure that not too much file data is purged, and that both types of purging have a chance to happen. The archiver stops purging files if there is no progress in purging tables. It is good practice to adjust the table threshold to match the archiver threshold. The archiver threshold depends on the type of appliance and whether failover is configured or not. The default value of the table threshold is 90% without HA and 80% for HA pair, which matches the lowest of all archiver thresholds. This value results in balanced data removal in most cases. To change the default value of the table purging threshold, see Changing the Database Purging Threshold.

From the Administration > System Settings > Database Table Retention page, you can select the number of days or disable retention for:

  • File Transfer History Settings
  • Database contents, by table category (indexed Alert, Report, and Statistic Tables)

The available numbers of days to select varies for different table categories. Settings made in this tab override the default settings for this appliance model.

Disabling retention settings immediately purges the data collected for that file or database category.

To set the retention time period for raw and index data for any appliance (LX, MX, and ST), see Creating a New Retention Rule.

It is good practice to simplify retention settings by using consistent retention durations throughout to avoid confusion. However, you can use data retention settings to ensure the data you need most is retained longer than data you don't need as much. For example, if you need Authentication data and don't need SEC Deny IP data, set Authentication to a higher duration and SEC Deny IP to a lower duration.

For archival purposes, confirm that the appliances are receiving log data. For details, see Forwarding Logs to Other Appliances (Routing) and Log Source Management.

For each table category, the Database Table Retention tab displays the Table Category, Duration in number of days, the number of Database Entries, and Data Size for each category.

To deactivate data or file retention, instead of letting the appliance purge files as disk space fills up, you may select Disabled from the Duration drop-down menu for that file or database table category.

Warning: Selecting Disabled sets the file retention period to zero, and the data is immediately purged from the appliance for that file or database category.

Click Update to save your changes.