Technical Tip: CSTN 00054 - FortiSOAR Performance Benchmarking for v6.4.0

[nmathur]

Description
This article describes the performance benchmark tests conducted in Fortinet labs.
The performance benchmarking tests were performed on FortiSOAR™ version 6.4.0 Build 1555.
Solution

The objective of this performance test is to measure the time taken to create alerts in FortiSOAR™, and complete the execution of corresponding playbooks on the created alerts on a single-node FortiSOAR™ appliance and a cluster setup of FortiSOAR™.

The data from this benchmark test can help you in determining your scaling requirements for a FortiSOAR™ instance to handle the expected workload in your environment.

Environment

FortiSOAR™ Virtual Appliance Specifications

Component	Specifications
CPU	8 CPUs
Memory	16 GB
Storage	250 GB virtual disk running on top of Samsung SSD 360 Pro model attached to VMware ESX server

Operating System Specifications

Operating System	Kernel Version
CentOS 7	3.10.0-1062.9.1.el7.x86_64

Pre-test Conditions

At the start of each test run -

• The test environment contained zero alerts.
• The test environment contained only the FortiSOAR™ built-in connectors such as IMAP, Utilities, etc.
• The system playbooks were deactivated and there were no running playbooks.
• The playbook execution logs were purged.

Details of the FortiSOAR™ Performance Benchmarking Test

The test was executed using an automated testbed that initiated HTTPS calls per clock tick (X alert API calls triggered at once) which created alerts in FortiSOAR™ and then triggered a playbook for each alert created. Steps are as follows:

1. The alerts were created using JMeter to simulate parallel invocation of the API - ‘/api/3/alerts/’.
2. When an alert was created, a post-create playbook is triggered which performs the following steps: 
   • Declare variables using the Set Variable Step
   • Extract artifacts from the source data of the alert using the “FSR: Extract Artifacts from String” action of the "Utilities" connector.
   • Add the extracted artifacts in the “Closure Notes” field of the alert.
   • Update the status of the alert to “Closed”.

Setup

The test is invoked in two different setups:

•  A single node FortiSOAR™ machine : This machine is a standalone machine for FortiSOAR™ and all calls are directly ingested by this machine.
• A High Availability (HA) two-node cluster with an HA Proxy: This setup is a cluster of two FortiSOAR™ machines which are joined in the Active-Active state using the FortiSOAR™ HA feature. This setup also contains an HA proxy, which acts as a load balancer. All the calls are directed to the HA proxy load balancer, which functions in a round-robin method to distribute the load of calls.

Observations

The data in the following tables outlines the number of alerts ingested in a clock tick, the total time taken to ingest those alerts, and the total time taken for all the playbooks triggered to finish execution.

Single Invocation Test run on a single-node FortiSOAR™ appliance 

Number of alerts created in FortiSOAR™	Total time taken to create all alerts in FortiSOAR™ (in seconds)	Total time taken to execute all Playbooks (in seconds)
25	4	2.5
50	5	3.33
100	7	6.23

Single Invocation Test run on a two-node Active-Active FortiSOAR™ cluster 

Number of alerts created in FortiSOAR™	Total time taken to create all alerts in FortiSOAR™ (in seconds)	Total time taken to execute all Playbooks (in seconds)
25	2	1.39
50	3	1.88
100	5	3.91

Sustained Invocation Test

In the sustenance test conducted on a two-node Active-Active FortiSOAR™ cluster, we could ingest 200 Alerts every 20 secs over 12 hours and observed that 381795 alerts were generated and corresponding playbooks successfully completed.

In the sustenance test conducted on a single node machine, we could ingest 100 Alerts every 20 secs over 12 hours and observed that 225257 alerts were generated and corresponding playbooks successfully completed.

Conclusion

Based on this test, we conclude that FortiSOAR™ could process an average of 18771 alerts in an hour in a single node and 31297 alerts in an hour in a two-node Active-Active FortiSOAR™ cluster. This includes the creation of alerts and running corresponding playbooks to process the alerts.

Notes

In a production environment the following factors might vary, which could affect the observations:

1. The size of alert data.
2. The number of playbooks that are being executed in parallel for each alert (ex, system playbook for notification or triage/investigate playbooks).
3. The number of steps in each playbook.
4. The network bandwidth especially for outbound connections to applications such as VirusTotal.