Description
This article describes how event data is processed in FortiSIEM.
Solution
FortiSIEM has a shared in-memory buffer for temporarily storing events for real time processing. The buffer is single-writer-multiple-reader circular buffer. In version 5.x the shared memory is called "admin". To check the shared memory segments, enter the following command:
ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x0052e2c1 0 postgres 600 291446784 23
0x4303a086 32769 root 600 300408 0
0x000003f2 65538 admin 777 536871240 8
PhParser is the only writer for the shared memory buffer. On the other hand, there are multiple readers;
- phDataManager (index the data and store it to NFS)
- phRuleWorker (partial rule processing – feeds partial results to ruleMaster)
- phReportWorker (partial report processing of inline reports for showing in dashboard –feeds partial results to ReportMaster)
- phIdentityWorker (partial identity and location processing – feeds partial results to IdentityMaster)
While data is written to NFS by DataManager, Rules are getting processed by RuleWorker and RuleMaster.
There is a natural back pressure built into the system: If NFS is slow, then DataManager is slow, shared in-memory buffer gets full and parser cannot write new events and collectors can not send data to cloud in real time which translates in lag.
FortiSIEM (or any such system) will not be able to function with a slow I/O datastore because it is designed as a "do-not-lose-events" system.
