Shredding LogLogic LMI Event Data

To comply with the GDPR regulations, LogLogic LMI provides a CLI utility to shred selected data.

You can use the llshred utility on the result of an Advanced Search query to shred a list of events from the result. If the utility is run with dryRun=true, the log data is only processed. However, with dryRun=false, the log data is prefixed with the string LLSHREDDER and all characters are replaced with X. The number of Xs is equal to the number of bytes in the original message.

For example, the following log entry includes 55 characters: 

llfeed,03/23/2018 15:28:06.584,tcp,10252,19208,attitude

The utility replaces the log entry with the string LLSHREDDER followed by 45 Xs: 

LLSHREDDERXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

If an event is removed or its length changed, the references to the subsequent events become invalid. Therefore, the utility shreds the log events rather than deleting them. To shred events ingested into the appliance after running the utility, you must run the utility again.

Procedure

  1. Back up the data on your appliance.
    Caution: The only way to recover shredded events is by restoring a backup. Therefore, you must take a backup before running the utility.
  2. Run an Advanced Search query for the events you want to shred. The query must include at least the sys_eventKey column in the projection. The sys_eventKey column can be in any position in the projection. For example: 
    SQL Query: 
select sys_eventKey, sys_body from system
where sys_collectIP='192.168.56.1' and sys_body contains 'attitude'

EQL query:
use system |'attitude'| sys_collectIP='192.168.56.1'| columns sys_eventKey, sys_body
  3. Save the search result as a .csv file.
  4. Copy the .csv file to the appliance using a file transfer tool such as SCP.
  5. From the CLI, run the llshred Command on the data in the .csv file to shred the data.

Impact of llshred Utility on Search and Reports

Advanced Search
If you specify a search term using Regex or contains, no result is returned. When you use any of the following fields (exclusively) in a WHERE clause or in the source filter of a data model, the search result returns the shredded log events:
  • sys_eventTime
  • sys_collectIP
  • sys_deviceType
  • sys_device
  • sys_collectorDomain
  • sys_eventKey
  • sys_filename
The data before shredding is stored in the cache as long as the Advanced Search tab is open. To verify if the data has been shredded, you must close the tab in which the search was executed, so that the cache is cleared.
Index Search, Real-time Reports
Index search and real-time reports return the shredded logs (containing the Xs) in the results even after running the llshred utility.
Regex Search
Regex search does not return the shredded logs (containing the Xs) in the results after running the llshred utility.