Database Table Retention Tab

When is data purged?

Data stored in the appliance database is purged in the following scenarios:

• On expiration: All database data eventually expires and is deleted. The expiration is specified in the Duration field on the Administration > System Settings > Database Table Retention page.
• On emergency: When the disk usage of /loglogic reaches a threshold value defined in LxDiskUsagePercent. See Changing the Database Purging Threshold. The same parameter is used by the scheduler for both LogLogic LX Appliances and LogLogic ST Appliances.

The Scheduler

The scheduler is involved in data purging and handles emergency purging of MySQL tables. A table with the oldest data is removed first, regardless of log type or table type. However, eventually all event data tables are purged if disk usage for /loglogic does not fall below the threshold setting value.

The scheduler deletes all data stored in the database tables and the engine_archive process does not start performing emergency purging as long as the scheduler is still purging database tables.

The archiver

The archiver handles purging of raw and index data. A file with the oldest expiration time is deleted first. To set the retention time of raw logs and index data, see Data Retention Rules.

The archiver begins purging raw and index data when the scheduler sets a flag to notify the archiver that the scheduler has deleted all the event data it can from the database. If the disk usage of /loglogic is still above the emergency purge threshold and the scheduler sets the flag, engine_archive begins the deletion it is responsible for. It is best practice to adjust the emergency purging threshold of the scheduler to match the emergency purging threshold of the engine_archive process. The archiver threshold depends on the type of appliance and whether failover is configured or not. The default value of the table threshold is 90% without HA and 80% for HA pair, which matches the lowest of all platform models for this threshold. This value results in balanced data removal in most cases. To change the default value of the table purging threshold, see Changing the Database Purging Threshold.

Retention settings

From the Administration > System Settings > Database Table Retention page, you can select the number of days or disable retention for:

• File Transfer History Settings
• Database contents, by table category (indexed Alert, Report, and Statistic Tables)

The available numbers of days to select varies for different table categories. Settings made in this tab override the default settings for this appliance model.

Disabling retention settings immediately purges the data collected for that file or database category.

To set the retention time period for raw and index data for LogLogic LX Appliance, LogLogic MX Appliance, and LogLogic ST Appliances, see Creating a New Retention Rule.

It is good practice to simplify retention settings by using consistent retention durations throughout to avoid confusion. However, you can use data retention settings to ensure the data you need most is retained longer than data you don't need as much. For example, if you need Authentication data and don't need SEC Deny IP data, set Authentication to a higher duration and SEC Deny IP to a lower duration.

For archival purposes, confirm that the appliances are receiving log data. It is best practice to have a copy of the older data, if you reduce retention settings. For details, see Forwarding Logs to Other Appliances (Routing) and Log Source Management.

For each table category, the Database Table Retention tab displays the Table Category, Duration in number of days, the number of Database Entries, and Data Size for each category.

To deactivate data or file retention, instead of letting the appliance purge files as disk space fills up, you may select Disabled from the Duration list for that file or database table category.

Click Update to save your changes.