Technical Tip: Guide to FortiSOAR™ Tuning

[nmathur]

Description
This article provides you with information on how you can tune your FortiSOAR™ (FSR) system to ensure that your system is running smoothly and with optimum performance.
Solution

Updating the SSL certificates

Replace the default self-signed certificates of the FortiSOAR™ web server with the signed certificates of your organization using the following procedure:

1. SSH to your FortiSOAR™ VM and login as a root user. 
2. To deploy your certificate, type the following command:
   # csadm certs --deploy 
   You must then specify the following at the prompt:
   The complete path of the private key file of your ssl certificate.
   The complete path to the crt file of your ssl certificate.
   Note: Your SSL certificate file must be in the .crt and .key format. 

Setting up System Monitoring

You should enable the "System Monitoring" playbook included with FortiSOAR™ to effectively monitor various FortiSOAR™ system resources such as CPU, Disk Space and Memory utilization, and the statuses of various FortiSOAR™ services. To know more about setting up thresholds, notifications, and schedules, see the System Monitoring: Setting up thresholds, schedules, and notifications article.

Setting up purging for Playbook logs

FortiSOAR™ persists each workflow step inputs, outputs and error details for providing granular details of each action run, which is very useful for subsequent analysis and debugging. However, this generate large volumes of data that might not be useful after some point of time. One must, therefore, configure a purge schedule for these logs to keep the database and disk usage for these logs constant over time. The retention period of playbook logs should preferably be only a few weeks. You can schedule purging, on a global level, for executed playbook logs. For the procedure for enabling and scheduling purging, see the System Configuration chapter in the "Administration Guide."

Setting up purging for Audit logs

FortiSOAR™ audits every login, logout, record create, update, delete of records and other important activity on the system. The audit logs might be useful for only a few years. Therefore, you must schedule a purging for audit logs as per your organization's retention policy. You can schedule purging, on a global level, for audit logs. For the procedure for enabling and scheduling purging, see the System Configuration chapter in the "Administration Guide."

Configuring High Availability or Disaster Recovery options

There are multiple ways to ensure that FortiSOAR™ data is backed up and available for recovery in case of a disk or VM failure. Some of the options follow:

1. Configuring FortiSOAR™ with an external PostgreSQL database if your organization has PostgreSQL racks centrally configured and managed.
2. Scheduling backup of the FortiSOAR™ database and configuration files to a shared drive using the backup CLI.
3. Configuring FortiSOAR™ cluster of 2 or more nodes.

For more information, see the Backing up and Restoring FortiSOAR™ chapter and the  High Availability support in FortiSOAR™ chapter in the "Administration Guide." 

Setting up a proxy server to service all requests from FortiSOAR™

The FortiSOAR™ instance would need access to the following endpoints on the public internet:

• For upgrading FortiSOAR™ and installing connectors: update.cybersponse.com.
• For installing python dependencies for connectors: pypi.python.org for installing connector python dependencies.
  Note: There is a parallel python repository also on update.cybersponse.com that can be used with some configuration if your organization does not approve pypi.
• For synchronization of FortiSOAR™ license details: globalupdate.fortinet.net.
• For accessing any SaaS or API endpoint that you have configured, for example VirusTotal, and to which you require to be connected.

Ensure that these endpoints are open from the organizations proxy, and also specify the proxy details to the FortiSOAR™ node during the FortiSOAR VM configuration wizard that is launched on the first boot of  the FortiSOAR™ appliance. You can also update the proxy details later using the csadm proxy CLI option, or from the FortiSOAR™ UI Setting > System Configuration > Environment Variables tab. 

Setting up NTP sync for the FortiSOAR™ server

If the time on the FortiSOAR™ server is not correct, you might see issues such as ingestion workflows not pulling the latest data from an external source. It is highly recommended to keep the time in sync with an NTP server. Edit the /etc/ntp.conf file to add details of a valid ntp server(s). For a list of common NTP servers, go to https://www.ntppool.org/en/. In case your FortiSOAR™ VM does not have access to the internet, then you must edit the /etc/ntp.conf to add details of a valid ntp server within your datacenter.

Configuring SMTP for FortiSOAR™

Configure the SMTP connector to receive any system or email notifications, including requests for resetting passwords. The SMTP connector is part of a number of pre-installed connectors or built-ins that are included with FortiSOAR™. By default, the SMTP connector is configured to use FortiSOAR™ appliance as an SMTP relay server. You must point it to a production SMTP server in your organization. 

Important: When you configure the SMTP connector, ensure that you select the Mark As Default Configuration option for the configuration that will be used for sending system notifications.

Backing up the data encryption keys

Encryption keys are used to encrypt data in FortiSOAR™. When you install FortiSOAR™ for the first-time default encryption keys are added, which are unique per instance; therefore, you do not need to change the encryption keys. 

You should copy the encryption keys from the /opt/cyops/config/cyops-api/application.conf file and store them securely in a Password Manager or Vault.

Once you encrypt your production data in FortiSOAR™ using the encryption keys, you should not change those keys again; since if your encryption keys are changed, this might result in the loss of previously encrypted production data. If you do require to change the encryption keys, then contact FortiSOAR™ Support.

Other Tunables

Tuning the number of playbook workers for running more workflows in parallel

The default number of playbooks that can run in parallel on a FortiSOAR™ instance is equal to the number of vCPUs on the machine. However, in most cases, it can also handle an increased number of workers. For example, a machine with 8 vCPUs can run upto 12 playbook workers in parallel. To change this value:

1. Add the following parameter to the /etc/celery/celeryd.conf file:
   CELERYD_OPTS="--concurrency=12"
2. Restart celeryd using the following command:
   systemctl restart celeryd
3. Monitor the CPU usage of the system over a day. If the CPU usage is very high, thev revert the settings to the default value.

Changing Elaticsearch memory setting

FortiSOAR™ uses Elaticsearch for faster searching capabilities. By default, it is configured to use 4 GB of RAM. If there are too many records or any very heavy records (such as large files uploaded) created per day on the system, it might crash with "out of memory" errors. To fix this, you must increase the memory allocated to Elasticsearch:

1. Change the following entry in /etc/elasticsearch/jvm.options to a higher value based on memory available on your server:
   -Xms4g 
   -Xmx4g
2. Restart Elasticsearch using the following command:
   systemctl restart elasticsearch

Changing Postgres worker memory

When the primary data in the system becomes large (eg, over million alerts) and you notice that the system is slow to respond. The slowness could be caused due to database queries taking longer with the increased database size. You can fine tune this behavior by increasing the following Postgres settings based on the available free memory on the system:

1. Increase the shared buffer and worker memory in the /var/lib/pgsql/12/data/postgresql.conf file:
   shared_buffers = 2048MB
   work_mem = 16MB
2. Restart Postgres using the following command:
   systemctl restart postgresql-12