Description
This article describes how to reduce memory usage by reducing some processes in FortiOS such as the IPS engine, WAD and SSL VPN which spawn a child process for each CPU core.
Each of the spawned child processes will have some memory allocated to it regardless of the traffic load.
For some units with multi-core CPUs and less amount of memory, mainly 'E' series such as 30E, 60E and 90E this can create an issue with memory usage.
Scope
FortiGate.
Solution
To solve memory usage issues, it is recommended to decrease the number of instances spawned by the aforementioned processes.
Confirm the number of instances spawned:
diagnose sys top-summary <- For firmware above 6.4 refer to the command 'diag sys top' mentioned below in this article.
CPU [| ] 4.8%
Mem [||||||||||||||||| ] 44.0% 836M/1866M
Processes: 20 (running=1 sleeping=130)
PID RSS CPU% ^MEM% FDS TIME+ NAME
* 144 163M 0.0 8.7 390 00:18.49 ipsmonitor [x6]
217 66M 0.0 3.6 197 00:20.76 wad [x9]
In the example above, there are x6 instances of the IPS engine and x9 of WAD, which all consume 8.7+3.6 = 12.3% of the memory while this unit processes almost no traffic at all.
Note that the command 'diagnose sys top-summary' is not supported in FortiOS 6.4 or later, however, 'diagnose sys top <delay> <number of processes>' will return the list of processes/workers. Be aware that the 'diagnose sys top' command is limited to a maximum of 99 entries.
More info can be found in Technical Tip: Deprecated of command '# diagnose sys top-summary'.
diagnose sys top 5 99
Run Time: 33 days, 4 hours and 16 minutes
0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 1863T, 463F
cw_wtpd 196 R 0.4 0.5
newcli 6591 R 0.4 0.3
ipsengine 6826 S < 0.0 4.5
ipsengine 6828 S < 0.0 4.3
ipsengine 6829 S < 0.0 4.3
ipsengine 6827 S < 0.0 4.3
.....
It is also possible to run 'diag sys top | grep <name>' to find out the number of instances spawned:
diag sys top 5 99 | grep wad
wad 1988 S 0.0 1.3 0
wad 2021 S 0.0 1.3 0
wad 2016 S 0.0 1.1 0
wad 12010 S 0.0 1.1 0
In the example above, there are x4 instances running for wad.
By using the commands below, it is possible to decrease the number of those instances spawned by the parent processes:
config system global
set wad-worker-count 2
end
config ips global
set engine-count 2
end
Now using the same command, to confirm:
diagnose sys top-summary
CPU [| ] 4.8%
Mem [||||||||||||||| ] 38.0% 722M/1866M
Processes: 20 (running=1 sleeping=127)
PID RSS CPU% ^MEM% FDS TIME+ NAME
* 284 102M 0.0 5.5 236 00:13.71 ipsmonitor [x4]
323 38M 0.0 2.0 132 00:11.17 wad [x7]
Verify that both have decreased their spawned child processes.
There will be 2 WAD workers and 2 IPS engine instances.
There are some instances that cannot be disabled, like IPS monitor and IPS helper or WAD manager.
That is why the x4 IPS monitor and x7 WAD are still visible.
However, notice the memory decreased to 7.5% for both processes.
Almost 5% additional memory was released.
An alternative approach to list the number of running workers would be to use the 'fnsysctl ps' command to list all processes running on the FortiGate.
fnsysctl ps
There are further methods to view the number of running processes for a given name:
For example, the number of PIDs (process IDs) can be listed with the command 'diag sys process pidof <process name>' for each process name.
diagnose sys process pidof miglogd
diagnose sys process pidof sslvpnd
diagnose sys process pidof scanunitd
More specifically the number of workers can be listed for IPS, WAD and miglogd as follows:
IPS:
diagnose test application ipsmonitor 1
pid = 1943, engine count = 4 (+1) <- There is one master process (ipsmonitor) and 4 workers (ipsengine) running.
0 - pid:1996:1996 cfg:1 master:0 run:1
1 - pid:7038:7038 cfg:0 master:1 run:1 <- master process master:1.
2 - pid:7032:7032 cfg:0 master:0 run:1
3 - pid:7033:7033 cfg:0 master:0 run:1
4 - pid:7034:7034 cfg:0 master:0 run:1
WAD:
diagnose debug reset
diagnose debug enable
diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=1145 diagnosis=yes.
Process [1]: type=dispatcher(1) index=0 pid=1179 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [2]: type=worker(2) index=0 pid=1180 state=running <- worker process 1.
diagnosis=no debug=enable valgrind=supported/disabled
Process [3]: type=worker(2) index=1 pid=1185 state=running <- worker process 2.
diagnosis=no debug=enable valgrind=supported/disabled
Process [4]: type=algo(3) index=0 pid=1176 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [5]: type=informer(4) index=0 pid=1175 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [6]: type=user info(5) index=0 pid=1177 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [7]: type=debug(8) index=0 pid=1174 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
miglogd:
diagnose test application miglogd 15
Main miglogd: ID=0, children=4, active-children=4 <- miglogd has 4 workers running.
ID=1, duration=10286129.
ID=2, duration=6.
ID=3, duration=6.
ID=4, duration=6.
The number of different spawned daemons can be reduced.
The numbers used in the example below are recommended on low-end devices like 60E.
These can be experimented with and adjusted as required.
config system global
set miglogd-children 1
set sslvpn-max-worker-count 2
set wad-worker-count 2
set scanunit-count 2
end
config ips global
set socket-size [integer, 0-512] <- IPS socket buffer size. Max and default value depend on available memory. Lower value reduces memory usage.
set engine-count [integer, 0-255] <- Number of IPS engines running. The default value of 0, FortiOS sets the number to optimize performance depending on the number of CPU cores. Reduce it to the number of cores the FortiGate box has.
set database [regular|extended] <- Regular protects against the latest common and in-the-wild attacks. Extended includes protection from legacy attacks.
end
After changing the engine, database, and socket size, restart the IPSEngine using the following command:
diag test app ipsmonitor 99
To reduce Spawn Processes in an HA-cluster, this process must be performed on both the Master and Slave Units because it is a hardware-specific configuration and will not sync automatically.
For example, to reduce the WAD worker count on HA-cluster, configure the following commands on both HA-cluster units:
config system global
set wad-worker-count 2
end
