Some VDOM stop sending traffing

[Virgule59]

Hi, 

I've a strange behevior with my fortigates 100E V7.0.12 cluster HA A-P, 5 Vdom in two vclusters.

3 vdom in vcluster 1 (root included) and 3 vdom in vcluster 2,

Forti1 is master, Forti 2 is slave

Forti1 is master for vcluster 1 and slave for vcluster2

Forti2 is master for vcluster 2 and slave for vcluster1  

Since two week on sunday and only on sunday at 00:00AM to 00:00PM, vdom in vcluster1 seems to stop accepting traffic in and out. External interface is ok and pinggable, but i'm unable to access web server behind. On internal interface routing is not working, VPN are up but not sure they accept traffic.

Try to disable DST and change fortiguard ip but that dont works.

Reboot Forti1 and forti2. Shutdown Forti1 and restart it. no change.

vdom in vcluster 2 seems to work as i'm connecting in vpn ssl to one VDOM and can remote admin forti01 from cli and ex ha manage.

Can you help me ?

Regards

[KumarV]

Hello @Virgule59 

Since the issue is just happening on Sunday so it is most probably related to schedule option on the firewall polices. did you get some time to check that part? 

Otherwise, Your issue needs in depth troubleshooting. We have to run the sniffers and debugs in order to check where the traffic is being dropped.

Below are few of the useful commands to troubleshoot such issues:

#di sniffer packet any "host x.x.x.x and icmp" 4 0 l

For debugs

#di de reset

#di de di

#di de flow filter addr x.x.x.x

#di de flow filter proto 1

#di de flow trace start 100

#di de en

After you get the output disable the debugs

#di de di

#di de reset

Thank you

Verender

[Virgule59]

Hi Kumar,

Thank for your return, no schedule rules on the policy.

Sniffer give only internal network traffic, nothing come from gateway interface, and i'll know that in normal situation i've a lot of traffic on this interface  Really strange situation. Is there some sort of cron task on fortigate ?

id=20085 trace_id=87 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404d4, original direction"
id=20085 trace_id=88 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X254:1->X.X.X76:0) tun_id=0.0.0.0 from local. type=0, code=0, id=1, seq=0."
id=20085 trace_id=88 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404d4, reply direction"
id=20085 trace_id=89 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X75:1->X.X.X254:2048) tun_id=0.0.0.0 from INT_01N. type=8, code=0, id=1, seq=0."
id=20085 trace_id=89 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404a5, original direction"
id=20085 trace_id=90 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X254:1->X.X.X75:0) tun_id=0.0.0.0 from local. type=0, code=0, id=1, seq=0."
id=20085 trace_id=90 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404a5, reply direction"
id=20085 trace_id=91 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X76:1->X.X.X254:2048) tun_id=0.0.0.0 from INT_01N. type=8, code=0, id=1, seq=0."
id=20085 trace_id=91 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404d4, original direction"
id=20085 trace_id=92 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X254:1->X.X.X76:0) tun_id=0.0.0.0 from local. type=0, code=0, id=1, seq=0."
id=20085 trace_id=92 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404d4, reply direction"
id=20085 trace_id=93 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X75:1->X.X.X254:2048) tun_id=0.0.0.0 from INT_01N. type=8, code=0, id=1, seq=0."
id=20085 trace_id=93 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404a5, original direction"
id=20085 trace_id=94 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X254:1->X.X.X75:0) tun_id=0.0.0.0 from local. type=0, code=0, id=1, seq=0."
id=20085 trace_id=94 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404a5, reply direction"
id=20085 trace_id=95 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X76:1->X.X.X254:2048) tun_id=0.0.0.0 from INT_01N. type=8, code=0, id=1, seq=0."
id=20085 trace_id=95 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404d4, original direction"
id=20085 trace_id=96 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X254:1->X.X.X76:0) tun_id=0.0.0.0 from local. type=0, code=0, id=1, seq=0."
id=20085 trace_id=96 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404d4, reply direction"
id=20085 trace_id=97 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X75:1->X.X.X254:2048) tun_id=0.0.0.0 from INT_01N. type=8, code=0, id=1, seq=0."
id=20085 trace_id=97 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404a5, original direction"
id=20085 trace_id=98 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X254:1->X.X.X75:0) tun_id=0.0.0.0 from local. type=0, code=0, id=1, seq=0."
id=20085 trace_id=98 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404a5, reply direction"
id=20085 trace_id=99 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X76:1->X.X.X254:2048) tun_id=0.0.0.0 from INT_01N. type=8, code=0, id=1, seq=0."
id=20085 trace_id=99 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404d4, original direction"
id=20085 trace_id=100 func=print_pkt_detail line=5851 msg="vd-VDOM1:0 received a packet(proto=1, X.X.X254:1->X.X.X76:0) tun_id=0.0.0.0 from local. type=0, code=0, id=1, seq=0."
id=20085 trace_id=100 func=resolve_ip_tuple_fast line=5932 msg="Find an existing session, id-000404d4, reply direction"

Best regard 

[Virgule59]

Fortigate start working correctly past 00:00, i don't understand, this is really strange behavior