Description
This article describes how to automatically save WAD debug when a WAD process crashes.
Scope
FortiGate, ForitProxy.
Solution
In FortiOS and FortiProxy, the WAD process is responsible for Webproxy and Proxy-based inspection features.
The WAD process crashes when running the command 'diag debug crashlog read' or an event log in the system events.
When the WAD process crashes, sessions that are handled by the WAD process will be discarded and need to be re-established.WAD crash debug collection helps to investigate the WAD crashes by saving the WAD debug messages to the disk in the event of a WAD process crash.
This feature is useful in cases where a frequent wad crash is noticed. The debug output can be shared with Fortinet support for further investigation.
This feature can be enabled using CLI:
diag wad debug crash
enable <----- Enable saving debug messages when the process crashes.
disable <----- Disable saving debug messages when the process crashes.
max <----- Max save size. 1-10 (MB).
list <----- List a summary of crash logs.
read <----- Display saved debug log.
clear <----- Clear saved debug log.
Note:
Enable this feature only for troubleshooting purposes, for a specific time frame, and make sure to disable WAD crash debug collection after collecting required debugs.
Example :
- Enable WAD crash debug collection.
diag wad debug crash enable
- Wait until the next wad process crashes.
diag debug crashlog read
[p:641][s:16574]wad_cifs_profile_init(93): CIFS Profile 0x7f469be29308 [] of type 0 created
32: 2023-10-03 12:16:21 <00641> firmware FortiGate-3401E v6.4.7,build1911b1911,210825 (GA) (Release)
33: 2023-10-03 12:16:21 <00641> application wad
34: 2023-10-03 12:16:21 <00641> *** signal 11 (Segmentation fault) received ***
35: 2023-10-03 12:16:21 <00641> AVDB 1.00000(04/09/0018 18:07)
36: 2023-10-03 12:16:21 <00641> ETDB 1.00000(04/09/0018 18:07)
- List the crash log saved to disk.
diag wad debug crash list
proc_type:2 id:265217323
crash process : worker
crash times : 1
crash log name : wad_crash_2_265217323
crash log time : Tue Oct 3 12:16:21 2023
- Read the debug output of the crashed WAD process.
diag wad deb crash read
~~~~~~~~~~~ BEGIN file wad_crash_2_265217323 ~~~~~~~~~~
[p:641]wad_tcp_port_alloc(1349): alloc tcp_port=0x7f469be5d050
redirect 37 accepted 10.218.7.4:59854 -> 10.0.7.121:80 on 63
[p:641][s:16574]__wad_timer_list_alloc(147): tm_list=0x7f469dd84210 bucks=11 period=10000ms precise=1000ms active=0x7f469dd84268 inactive=0x7f469dd84278 last=0x7f469dd84308
[p:641][s:16574]__wad_timer_list_alloc(147): tm_list=0x7f469dd84330 bucks=11 period=10000ms precise=1000ms active=0x7f469dd84388 inactive=0x7f469dd84398 last=0x7f469dd84428
In the above example, a WAD worker with process ID 641 has crashed.
- To disable wad crash debug collection.
diag wad debug crash disable
