Description
This article discusses the configuration and considerations relating to Analytic and Archived Log retention periods.
Scope
FortiAnalyzer.
Solution
In FortiAnalyzer 5.4, retention periods can be set for Analytic Logs and Archived Logs.
Analytic Logs are logs stored in the SQL database of that ADOM and are available for reports. These logs are visible under 'Log View' in the different log sections and will be deleted when:
Archived logs are logs in raw format stored in the FortiAnalyzer. These can be used to rebuild a DB if necessary, and typically go back further than the Analytic Logs. They take up considerably less disk space, but cannot be used in reports, and are only visible in a simple format (with limited filtering options) under Log View -> Log Browse. The log files will be deleted when:
Important note:
Analytic Logs are logs stored in the SQL database of that ADOM and are available for reports. These logs are visible under 'Log View' in the different log sections and will be deleted when:
- The Analytic Log retention period is exceeded.
- The quota for the Analytic Log is exceeded.
Archived logs are logs in raw format stored in the FortiAnalyzer. These can be used to rebuild a DB if necessary, and typically go back further than the Analytic Logs. They take up considerably less disk space, but cannot be used in reports, and are only visible in a simple format (with limited filtering options) under Log View -> Log Browse. The log files will be deleted when:
- The log file is rolled and the newest entry in the log file exceeds the log retention period configured.
- The disk quota for Archived Logs is exceeded.
Important note:
If, for example, an archive retention period of 30 days has been configured, and the earliest entry in a log file is 45 days old, but the newest is only 25 days old, then this log file will be kept until the newest entry has reached an age of 30 days before it is deleted.
This can lead to some log files exceeding the archived retention period by significant margins. To prevent or limit this, enable scheduled log rolling under System Settings -> Device Log Settings.
ADOM quotas, and how much of the quota should be set aside for Analytics and Archive, can be configured under System Settings:
When ADOMs are enabled, on the left menu under All ADOMs there is the option Storage Info.
When ADOMS are disabled, there is a field Log Storage Policy in the System Information Widget. It can be edited to increase the disk quota, to set maximum ages for Archived and Analytic Logs, and to set a ratio of Analytic to Archived logs (like 60:40 -> 60% of the quota will be for Analytics, 40% for Archive).
ADOM quotas, and how much of the quota should be set aside for Analytics and Archive, can be configured under System Settings:
When ADOMs are enabled, on the left menu under All ADOMs there is the option Storage Info.
Considerations in setting a Disk Quota and setting the Analytic:Archive ratio
If ADOMs are enabled, consider that each ADOM will need a chunk of the entire available disk size.
If ADOMs are enabled, consider that each ADOM will need a chunk of the entire available disk size.
What logging volume does the FortiAnalyzer report? How much of that volume goes into each ADOM?
How long do logs need to be retained for reporting (Analytic retention)?
How long do logs need to be retained in general (Archive retention)?
Consider also if it is required to keep logs for a very long time, it might be an option to configure log upload on the FortiAnalyzer to have logs stored elsewhere (like an FTP server).
Analytic logs take up more disk space than Archived Logs; about 2-3 times as much for the same time period.
Calculation example:
One FortiGate logs to an ADOM with about 1 GB/day.
How long do logs need to be retained for reporting (Analytic retention)?
How long do logs need to be retained in general (Archive retention)?
Consider also if it is required to keep logs for a very long time, it might be an option to configure log upload on the FortiAnalyzer to have logs stored elsewhere (like an FTP server).
Analytic logs take up more disk space than Archived Logs; about 2-3 times as much for the same time period.
Calculation example:
One FortiGate logs to an ADOM with about 1 GB/day.
Management wants to have reports that cover a timeframe of a month so Analytic retention needs to be at least 30 days. Erring on the side of caution, set to 35 days.
Company policy dictates that records need to be retained 6 months so Archive retention needs to be at least 185 days. Erring on the side of caution, set to between 190 and 200 days.
This results in:
Archive requirements ~200GB (200 days x 1GB/day).
Analytic requirements ~100GB (35days x 1GB/day x 3).
Set the ADOM quota to at least 300GB, if possible a bit more; and set a ratio of about 35:65 to start with.
Also, set daily or weekly log rolling to ensure that archived log files are deleted in a timely fashion. Log rolling on a daily or weekly basis results in log files only containing information for a maximum of a day/week and thus ensures that the archive retention is not exceeded significantly.
Company policy dictates that records need to be retained 6 months so Archive retention needs to be at least 185 days. Erring on the side of caution, set to between 190 and 200 days.
This results in:
Archive requirements ~200GB (200 days x 1GB/day).
Analytic requirements ~100GB (35days x 1GB/day x 3).
Set the ADOM quota to at least 300GB, if possible a bit more; and set a ratio of about 35:65 to start with.
Also, set daily or weekly log rolling to ensure that archived log files are deleted in a timely fashion. Log rolling on a daily or weekly basis results in log files only containing information for a maximum of a day/week and thus ensures that the archive retention is not exceeded significantly.
Always calculate generously. Once the FortiAnalyzer deletes logs to enforce quotas, it is very difficult, if not impossible, to get them back.
Always monitor the quota in use, and if necessary provide more quota or shift the ratio between analytic and archive logs.
Related articles:
Technical Note: How to set log retention values in FortiAnalyzer
Technical Tip: How to estimate disk space needed for Archive and Analytics logs
Technical Tip: Extending disk space in FortiAnalyzer-VM/FortiManager-VM
