Description
This article describes few basic steps of troubleshooting traffic over the FortiGate firewall, and is intended as a guide to perform the basic checks on the FortiGate when a problem occurs and certain traffic is not passing.
All these steps are important for diagnostics. The data collected in this guide is needed when opening a TAC support case.
When parts of this data are not present, the assigned TAC engineer will likely ask for it.
The resolution of a case is considerably faster when this data is already attached in the case from the moment it is created.
Solution
When did this stop working? Did this work before?
No: For a new implementation, check once again if the setup guide was followed entirely, and nothing is missing
mention the setup guide that was followed (link) when opening a TAC case. Also attach the configuration backup so TAC can check what was configured.
Yes: What has changed since the time it was working?
-Standalone Upgrade: review the release notes for known problems. Check if the firewall can reach the internet, has DNS response (exec ping pu.bl.ic.IP, exec ping service.fortiguard.net)
- HA Upgrade: make sure both units are in sync and have the same firmware (get system status).
Check if the Master has access to both WAN and LAN (exec ping pu.bl.ic.IP, exec ping lo.ca.l.IP).
If not, check the routing table (get router info routing-table all; get router info routing-table detail x.x.x.x ). If it is needed to revert to a working version, make sure to collect all the logs or call us, otherwise the support can’t investigate or provide a possible cause.
To downgrade quickly to a previous firmware (the previous firmware version is kept in memory).
- Policy / inspection profiles changes: review the last change.
Does it work after reverting the previous changes?
Yes: The root cause is isolated. Fine tune the profiles/policy recently added/removed, so that it allows the traffic.
No: Check why the traffic is blocked, per below, and note what is observed. This log is needed when creating a TAC support case.
- Start with the policy that is expected to allow the traffic. Check the ID number of this policy.
- Make sure that the session from source to destination is matching this policy:(check 'policy_id=' in the output).
( Use the below command to do a policy lookup in CLI: diagnose firewall iprope lookup <source ip> <source port number> <destination ip> <destination port number> <protocol> <incoming interface>)
- If the session exists, then check the existing UTM profiles in that policy (AV, WebFilter, IPS, etc)
Remove them one by one until the traffic is restored.
- Check if the traffic flows ok when policy is changed to flow-based, instead of proxy-based.
Traffic logs, packet captures, and debug flow are the tools TAC use further to check that, always in conjunction with the configuration file (backup from GUI of “Global” context). Debug log may also be required.
When opening a TAC support case, attach them and in more complex scenarios, the traffic path is needed as well:
(ie: PC >> port1 (vlan 100, vdom TEST, policy 17) >> zone PROD >> vdom link TEST_to_PROD >> port9 (vlan 15, policy 413) >> internet port wa1 )
Traffic logs (logging must be enabled in policy) or Security logs (AV/Webfilter/IPS/etc.):
either the traffic is blocked due to policy, or due to a security profile. When available, the logs are the most accessible way to check why traffic is blocked. Logs also tell us which policy and type of policy blocked the traffic. Sometimes also the reason why. Attach relevant logs of the traffic in question. Export a small group of such logs from the logging unit (FortiGate GUI, FortiAnalyzer, FortiCloud, Syslog, etc).
Packet capture (sniffer):
On models with hardware acceleration, this has to be disabled temporarily in order to capture the traffic.
It is better captured from command line and log the SSH output.
Debug flow (firewall logic):
Common cases where traffic is not passing, and shown in debug flow for new sessions:
'Denied by forward policy check'.
'iprope_in_check() check failed, drop.
' reverse path check fail, drop'.
Common cases where traffic is allowed:
'sent to AV' / 'sent to IPS': traffic is sent to AV inspection / to flow-based inspection.
'Trying to offloading session from port1 to wan1' : user session is being copied from one interface to another interface.
'Find an existing session, id-0xxxxxxxx, reply direction': a session is already established and the traffic is flowing (possibly Layer7 problem - packet capture needed).
Debug log (snapshot of the system parameters at the time it is downloaded):
If Authentication and user groups are used in policies, check also this guide related articles below.
For SIP/VoIP issues, a packet capture (usually with 'port 5060' as filter) is absolutely necessary, along with the configuration (backup from GUI of 'Global' context).
Related Articles
Technical Tip: Selecting an alternate firmware for the next reboot
Troubleshooting Tip: FortiGate session table information
Technical Tip: Disabling NP offloading in security policy
Troubleshooting Tool: Using the FortiOS built-in packet sniffer
Troubleshooting Tip : debug flow messages "iprope_in_check() check failed, drop" - "Denied by forwar...
Technical Note: Details about FortiOS RPF (Reverse Path Forwarding), also called Anti-Spoofing
Technical Tip: How to download debug.log file
Technical Tip: Troubleshooting steps for blocked HTTP traffic when using TSAgent
https://docs.fortinet.com/document/fortigate/6.2.3/cookbook/54688/debugging-the-packet-flow
