Description
This article describes some specifics of the FortiAnalyzer Multi-Tier Architecture, and how to correctly configure the Log Storage Policy in “Collector(s) – Analyzer” aggregation scenario, in order to avoid quota issues and ensure optimal performance.
To handle high log rates from big number of logging devices, FortiAnalyzer allows log aggregation between multiple hardware appliances in 2 different operating modes.
For details regarding the FortiAnalyzer operating modes, and configuring the aggregation settings and log storage policies, refer to the corresponding sections of the FortiAnalyzer Administration Guide.
Scope
Solution
Example of multi-tier architecture
The requirement for this example would be to store all logs as archive for 365 days, and allow generating reports for 60 days back.
The collectors' purpose in this deployment is to generally act as a buffer.
By default, the units in 'Collector Mode' have no analytics SQL database, which allows their hardware resources to be utilized for receiving and storing big volumes of log data during the peak hours. Then the logs are transferred to the Analyzer at a configurable 'quiet' time of the day.
In case some log events need to be urgently reviewed before being sent to the Analyzer, the information is available on the Collectors under Log View/Log Browse.
After clarifying the roles, it should be clear that the main log storage and analytics tool in this scenario is the FortiAnalyzer unit in Analyzer mode.
It is important in this case, that the Analyzer's hardware is correctly selected in terms of storage size, and capability to handle intensive SQL database operations.
Then logically, the Log Storage Policy on the Analyzer should be the one configured for the required retention quotas (in this example 365 days Archive, 60 days Analytics).
The Collectors’ retention quotas should be configured shorter, or equal to the Analyzer’s.
To cover cases of Analyzer’s failure or hardware migration, the aggregation code is not checking if certain log file was already sent to the Analyzer.
So if the Collectors’ Archive retention is longer than the Analyzer’s, each aggregation will synchronize all files, regardless of the Analyzer’s storage policy.
In some cases, this may cause over-quota situation, like for example:
Collector:
Analyzer:
The Analyzer will eventually delete the oldest files, in order to enforce the quota, but every next aggregation will sync them over and over again, wasting network bandwidth, disk I/O and other hardware resources.
This article describes some specifics of the FortiAnalyzer Multi-Tier Architecture, and how to correctly configure the Log Storage Policy in “Collector(s) – Analyzer” aggregation scenario, in order to avoid quota issues and ensure optimal performance.
To handle high log rates from big number of logging devices, FortiAnalyzer allows log aggregation between multiple hardware appliances in 2 different operating modes.
For details regarding the FortiAnalyzer operating modes, and configuring the aggregation settings and log storage policies, refer to the corresponding sections of the FortiAnalyzer Administration Guide.
Scope
Solution
Example of multi-tier architecture
The requirement for this example would be to store all logs as archive for 365 days, and allow generating reports for 60 days back.
The collectors' purpose in this deployment is to generally act as a buffer.
By default, the units in 'Collector Mode' have no analytics SQL database, which allows their hardware resources to be utilized for receiving and storing big volumes of log data during the peak hours. Then the logs are transferred to the Analyzer at a configurable 'quiet' time of the day.
In case some log events need to be urgently reviewed before being sent to the Analyzer, the information is available on the Collectors under Log View/Log Browse.
After clarifying the roles, it should be clear that the main log storage and analytics tool in this scenario is the FortiAnalyzer unit in Analyzer mode.
It is important in this case, that the Analyzer's hardware is correctly selected in terms of storage size, and capability to handle intensive SQL database operations.
Then logically, the Log Storage Policy on the Analyzer should be the one configured for the required retention quotas (in this example 365 days Archive, 60 days Analytics).
The Collectors’ retention quotas should be configured shorter, or equal to the Analyzer’s.
To cover cases of Analyzer’s failure or hardware migration, the aggregation code is not checking if certain log file was already sent to the Analyzer.
So if the Collectors’ Archive retention is longer than the Analyzer’s, each aggregation will synchronize all files, regardless of the Analyzer’s storage policy.
In some cases, this may cause over-quota situation, like for example:
Collector:
Analyzer:
The Analyzer will eventually delete the oldest files, in order to enforce the quota, but every next aggregation will sync them over and over again, wasting network bandwidth, disk I/O and other hardware resources.
