|
The WAD process suffers a memory leak on FortiOS 7.0.8 and 7.0.9 in WAD processes with the 'user-info' type.
To confirm if the device is suffering from this issue, run the following diagnostic commands to show the total memory usage of the device:
# get sys stat
# get sys perf stat
# get hardware memory
# diagnose sys top-mem 99
# diagnose debug reset
# diagnose debug enable
# diagnose test app wad 1000
# diagnose test app wad 2500
# diagnose test app wad 803
# diagnose test app wad 2
# diagnose test app wad 3
# diagnose debug disable
# diagnose debug report
This will allow confirmation of the firmware version, as well as the current total memory usage and the kernel memory allocation.
Example outputs:
# get sys stat
Version: FortiGate-200E v7.0.8,build0418,221012 (GA.F)
# get sys perf stat
Memory: 4057460k total, 3063772k used (75.5%), 620072k free (15.3%), 373616k freeable (9.2%)
# get hardware memory
MemTotal: 4057460 kB
MemFree: 625160 kB
Cached: 962744 kB
Active: 2525552 kB <----- 2466,36 MB
Inactive: 270932 kB
Shmem: 502748 kB
Slab: 184460 kB
Most memory is allocated under 'Active' memory. The following command can show the user space processes using the active memory:
# diagnose sys top-mem 99
wad (236:( 1165711kB <--- 1138,38 MB
Here, a single WAD process uses approximately 1140 MB out of the total 3962 MB.
The process ID (PID) of this process is 236.
To determine which type this WAD process has, run the following:
# diagnose debug reset
# diagnose debug enable
# diagnose test app wad 1000
If the process type is 'user-info' as shown below...
Process [6]: type=user-info(5) index=0 pid=236 state=running
... then change of the context to the user-info process and show the memory usage type.
The command 'diagnose test app wad 2500' will switch to the user-info process context.
The values 803, 2 and 3 will dump the current memory stats for this user-info process.
# diagnose test app wad 2500
# diagnose test app wad 803
# diagnose test app wad 2
# diagnose test app wad 3
The issue can be identified if the memory usage in 'mmaped regions' from the 'diagnose test app wad 2' command is much higher than the memory in object 'wad_m_cmem_root' indicated by the 'diagnose test app wad 803' command.
For example:
# diagnose test application wad 2
process malloc info:
space in mmapped regions: 1406087168
# diagnose test application wad 803
cmem object stats:
id allocs frees reallocs avg_sz in_str active bytes max cmem object name
0 842340758 842310950 0 96 0 29808 2865986 3014515 wad_m_cmem_root
Workaround:
As a workaround, restart the WAD processes with the following command:
# diagnose test app wad 99
This can be automated with the 'config system auto-script' feature.
Solution:
The solution is to upgrade to FortiOS versions 7.0.10, 7.2.4, or above.
|
