How to test FortiSIEM IOPS storage performance

[yujames]

Description

This article describes the process to benchmarking NFS client or local storage performance. Several application issues can be traced back to poor IOPS. This guide helps evaluating the storage bandwidth available for FortiSIEM processes and ensuring the  sizing guide requirements are met on all FortiSIEM components relying on storage (Workers, Supervisor)

Solution

Login as ‘root’  on Super/worker via ssh. Install fio from default repositories:

yum install -y fio

Evaluate the current usage. The benchmark results could be misleading if the storage IO is already in use. The storage bandwidth should be tested, preferably when FortiSIEM services are stopped (phtools --stop ALL). Alternatively, the current IOPS usage must be taken in consideration.

1. IOPS usage for All in one FSM instances:
   1. iostat -d -p /dev/sdd 2 10 (consider the average blk read/write)
2. IOPS usage for FSM distributed setup (clustered) with NFS:
   1. nfsiostat /data. Performance is expressed in Kb Read/Write throughput. NFS operation size is different from local. Disk Operation size (IOPS) in nfsiostat kB/op indicates the amount of data per operation.

Benchmark storage:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=bw_test --filename=/data/test --bs=4k --iodepth=64 --size=2G --readwrite=randrw --rwmixread=30

--ioengine=libaio: Defines how the job issues I/O to the file. Libaio is the Linux  native  asynchronous  I/O method.

--direct=1: IO access memory caching (default OS behavior: page cache).

--name=bw_test : test (job) name.

--filename=/data/test: test file location. This was set to /data in this article, which is either the NFS or /dev/sdd mount point.

-bs=4k: Block size per operation.

--iodepth=96: blocks Queue length (this test is in async mode).

--size=2G: size of the test file.

--readwrite=randrw: Random Read/Write to simulate a real environment with random read/write operations.

--rwmixread=30 : Percentage of a mixed workload that should be read (most of the time, FortiSIEM operations are writes (hence the 70%) except when running heavy operations (such as searches, reports, etc).

Results:

1. read: IOPS=21.5k, BW=83.9MiB/s (87.0MB/s)(1023MiB/12197msec).
   1. IOPS=21.5k : average read I/Os performed per second.
2. write: IOPS=21.5k, BW=84.0MiB/s (88.1MB/s)(1025MiB/12197msec).
   1. IOPS=21.5k : average write I/Os performed per second.
3. lat (usec): min=125, max=9377, avg=728.78, stdev=381.83.
   1. latency: denotes the time from when fio created the I/O unit to completion of the I/O operation.