Help
FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
sha-1_FTNT
Staff
Staff

Created on ‎01-31-2017 08:57 AM Edited on ‎06-09-2022 09:24 PM By Anonymous

Article Id 194097

Technical Tip: Building a Layer-2 VPN with VxLAN over IPsec

Description

This article describes how to build a Layer-2 VPN between two FortiGates using VxLAN over IPsec.


Scope

Support for VxLAN over IPsec was added in FortiOS 5.4


Solution

 

Diagram

 

 

The following topology is used:
 
PC1(.1) – 192.168.100.0/24 - port2-[ FG1 ]-port1-(198.51.100.1) ====
                                                                    I
                                                   (Internet)       I VxLAN-over-IPsec tunnel

 

                                                                    I
PC2(.2) - 192.168.100.0/24 - port2-[ FG2 ]-port1-(203.0.113.2) =====

  

Design

 

  • Layer-2 VPN (aka Ethernet-VPN, EVPN) subnet 192.168.100.0/24 spans over two sites which are connected via a VxLAN-IPsec tunnel

     

  • A software switch is configured to bridge Ethernet frames between the local LAN and the VxLAN-IPsec tunnel

     

  • Ethernet frames forwarded to the remote site are encapsulated in UDP (VxLAN) then protected with IPsec (VxLAN over IPsec)

     

 

Limitations

 

  • In FortiOS 5.4, VXLAN is only supported as an encapsulation method within the configuration of an IPsec tunnel. Native VXLAN tunnel cannot be configured in FortiOS 5.4.
  • Support for native VxLAN is available as of FortiOS 5.6
  • IPsec processing cannot be hardware offloaded to NPU (NP6, NP4) when IPsec is configured with VxLAN encapsulation
  • FortiOS does not send back an ICMP “destination unreachable, fragmentation needed and DF set” to the source when an IP packet with the DF bit set and a size greater than the tunnel MTU cannot be forwarded inside the VxLAN-IPsec tunnel.

The workaround is to stop honoring the DF bit:

 

config system global

 

    set honor-df disable

 

end

 

 

  • FortiOS 5.4 VxLAN encapsulation within IPsec only supports forwarding untagged frames
The forwarding of tagged frames (802.1Q) is not supported.
More specifically, tagged frames with a size greater than the tunnel MTU are not forwarded/fragmented even if the inner IP packet has DF=0.
This kind of configuration is therefore NOT supported:

     config system interface

 

         edit "port2"                        // connected to a 802.1Q trunk port

 

             set vlanforward enable          // to forward Vlans undeclared in FortiOS

         next

 

         edit "VxLan-IPsec"

 

             set vlanforward enable

         next

 

     end

 

 

     config system switch-interface

 

         edit "soft_switch"

 

             set member "port2" "VxLan-IPsec"

 

               next
           end

Configuration

 

  

CLI configuration of FortiGate 1

 

 
#
# Port1 is the Internet-facing interface
# Port2 is the LAN interface which is placed in a soft-switch
#
 
config system interface
    edit "port1"
        set ip 198.51.100.1 255.255.255.0
        set alias "Internet"
    next
    edit "port2"
        set alias "LAN"
    next
end
 
 
#
# Configure an IPsec tunnel with VxLAN encapsulation
#
 
config vpn ipsec phase1-interface
    edit "VxLan-IPsec"
        set interface "port1"
        set peertype any
        set proposal aes128-sha1
        set encapsulation vxlan    // data packets are subject to VxLAN encapsulation prior to IPsec processing
        set remote-gw 203.0.113.2
        set psksecret someSecureKey
    next
end
 
config vpn ipsec phase2-interface
    edit "VxLan-IPsec"
        set phase1name "VxLan-IPsec"
        set proposal aes128-sha1
    next
end
 
#
# Define the soft-switch used to bridge traffic between the local LAN (port2) and the remote LAN (VxLan-IPsec)
#
config system switch-interface
    edit "soft_switch"
        set vdom root
        set member "port2" "VxLan-IPsec"
        set intra-switch-policy explicit   // (optional) mandates firewall policy lookup for bridged traffic
    next
end
 
#
# Allow traffic between the local LAN (port2) and the remote LAN (VxLan-IPsec)
# This is mandatory since “set intra-switch-policy explicit” was used in the soft-switch
#
 
config firewall policy
    edit 1
        set srcintf "port2"
        set dstintf "VxLan-IPsec"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
    next
    edit 2
        set srcintf "VxLan-IPsec"
        set dstintf "port2"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
    next
end
 
#
# Internet Access
#
 
config router static
    edit 1
        set gateway 198.51.100.253
        set device "port1"
        set comment "default-route to Internet ISP"
    next
end
 
 
//
// Alternative scenario: The LAN side could be a Vlan (sub-)interface.
//

 

config system interface
    edit "vlan1234"
        set interface "port2"
        set vlanid 1234
    next
end

 

config system switch-interface
    edit "soft_switch"
        set vdom root
        set member "vlan1234" "VxLan-IPsec"
        set intra-switch-policy explicit
    next
end

 

config firewall policy
    edit 1
        set srcintf "vlan1234"
        set dstintf "VxLan-IPsec"
        …
    next
    edit 2
        set srcintf "VxLan-IPsec"
        set dstintf "vlan1234"
        …
    next
end

 

This alternative configuration does not conflict with the fact that FortiOS only supports the forwarding of untagged frames. The 802.1Q tag has already been stripped off when a frame reaches the vlan1234 (sub-)interface. 802.1Q tagged frames are received on the physical interface (here, port2), not on the vlan1234 (sub-)interface.
 

 

 

CLI configuration of FortiGate 2

 


config system interface
    edit "port1"
        set ip 203.0.113.2 255.255.255.0
        set alias "Internet"
    next
    edit "port2"
        set alias "LAN"
    next
end
 
config vpn ipsec phase1-interface
    edit "VxLan-IPsec"
        set interface "port1"
        set proposal aes128-sha1
        set encapsulation vxlan
        set remote-gw 198.51.100.1
        set psksecret someSecureKey
    next
end
 
config vpn ipsec phase2-interface
    edit "VxLan-IPsec"
        set phase1name "VxLan-IPsec"
        set proposal aes128-sha1
    next
end
 
config system switch-interface
    edit "soft_switch"
        set member "port2" "VxLan-IPsec"
        set intra-switch-policy explicit
    next
end
 
config firewall policy
    edit 1
        set srcintf "port2"
        set dstintf "VxLan-IPsec"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
    next
    edit 2
        set srcintf "VxLan-IPsec"
        set dstintf "port2"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
    next
end
 
config router static
    edit 1
        set gateway 203.0.113.254
        set device "port1"
        set comment "default-route to Internet ISP"
    next
end

  

Verification

 

 
Verify that PC1 can ping PC2

 

root@PC1:~# ping -c 5 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=10.1 ms
64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=8.45 ms
64 bytes from 192.168.100.2: icmp_seq=3 ttl=64 time=17.9 ms
64 bytes from 192.168.100.2: icmp_seq=4 ttl=64 time=8.28 ms
64 bytes from 192.168.100.2: icmp_seq=5 ttl=64 time=15.3 ms
 
--- 192.168.100.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4010ms
rtt min/avg/max/mdev = 8.282/12.042/17.964/3.905 ms
 
 
Verify the IPsec tunnel

 

FG1 # diag vpn tunnel list
list all ipsec tunnel in vd 0
------------------------------------------------------
name=VxLan-IPsec ver=1 serial=3 198.51.100.1:0->203.0.113.2:0
bound_if=3 lgwy=static/1 tun=intf/0 mode=auto/1 encap=VXLAN/6 options[0006]=edg
encap-addr: 198.51.100.1->203.0.113.2
proxyid_num=1 child_num=0 refcnt=14 ilast=16 olast=317 auto-discovery=0
stat: rxp=10 txp=11 rxb=1784 txb=1022
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=843
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=VxLan-IPsec proto=0 sa=1 ref=2 serial=1
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0
  SA: ref=3 options=2e type=00 soft=0 mtu=1390 expire=42473/0B replaywin=2048 seqno=c esn=0 replaywin_lastseq=0000000b
  life: type=01 bytes=0/0 timeout=43147/43200
  dec: spi=e1677820 esp=aes key=16 199f8427aaede3b3ae875320a76ce4c7
       ah=sha1 key=20 1698424168fb6834e68dfc57e927ca77c60da93c
  enc: spi=a8d52b6c esp=aes key=16 5e78151deb161b9c541b7fa2938b2a35
       ah=sha1 key=20 e94b8f491e5c519c5ea19519665d161421019c00
  dec:pkts/bytes=10/1284, enc:pkts/bytes=11/1968
 
 
Verify the soft-switch forwarding table

 

FG1 # diagnose netlink brctl name host soft_switch
show bridge control interface soft_switch host.
fdb: size=2048, used=4, num=4, depth=1
Bridge soft_switch host table
port no device  devname       mac addr            ttl     attributes
  2     4       port2         02:09:0f:00:02:02    2        Hit(2)      ß The MAC address of PC1 (local)
  1     21      VxLan-IPsec   02:09:0f:00:06:03    2        Hit(2)      ß The MAC address of PC2 (remote)
  2     4       port2         02:09:0f:00:02:03    0       Local Static
  1     21      VxLan-IPsec   d2:ff:26:ba:45:9c    0       Local Static

 

 

Troubleshooting

 

 
Verify that ARP traffic can flow through the VxLAN-IPsec tunnel
 
FG1 # diag sniffer packet any 'host 192.168.100.2 and arp' 4
interfaces=[any]
filters=[host 192.168.100.2 and arp]
12.601491 VxLan-IPsec in arp who-has 192.168.100.1 tell 192.168.100.2
12.601508 port2 out arp who-has 192.168.100.1 tell 192.168.100.2
12.601882 port2 in arp reply 192.168.100.1 is-at 2:9:f:0:2:2
12.601888 VxLan-IPsec out arp reply 192.168.100.1 is-at 2:9:f:0:2:2
 
4 packets received by filter
0 packets dropped by kernel
 
 
Verify the debug flow when PC1 attempts to ping PC2

 

FG1 # diag debug flow filter clear
FG1 # diag debug flow show function-name enable
show function name
 
FG1 # diag debug flow show iprope enable
show trace messages about iprope
 
FG1 # diag debug flow filter proto 1
FG1 # diag debug flow filter addr 192.168.100.2
FG1 # diag debug flow show console enable
show trace messages on console
 
FG1 # diag debug flow trace start 1000
FG1 # diag debug enable
 

## ICMP echo-request from PC1 to PC2

 

id=20085 trace_id=51 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 192.168.100.1:2871->192.168.100.2:2048) from port2. type=8, code=0, id=2871, seq=1."

id=20085 trace_id=51 func=init_ip_session_common line=4944 msg="allocate a new session-00002434"
id=20085 trace_id=51 func=iprope_dnat_check line=4659 msg="in-[port2], out-[]"
id=20085 trace_id=51 func=iprope_dnat_check line=4672 msg="result: skb_flags-06000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=20085 trace_id=51 func=iprope_fwd_check line=636 msg="in-[port2], out-[VxLan-IPsec], skb_flags-06000000, vid-0"
id=20085 trace_id=51 func=__iprope_check line=2049 msg="gnum-100004, check-ffffffffa001e70e"
id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-100004 policy-1, ret-matched, act-accept"
id=20085 trace_id=51 func=__iprope_user_identity_check line=1648 msg="ret-matched"
id=20085 trace_id=51 func=__iprope_check line=2049 msg="gnum-4e20, check-ffffffffa001e70e"
id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
id=20085 trace_id=51 func=__iprope_check_one_policy line=1823 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept"
id=20085 trace_id=51 func=__iprope_check line=2068 msg="gnum-4e20 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000"
id=20085 trace_id=51 func=__iprope_check_one_policy line=2020 msg="policy-1 is matched, act-accept"
id=20085 trace_id=51 func=__iprope_check line=2068 msg="gnum-100004 check result: ret-matched, act-accept, flag-08010000, flag2-00004000"
id=20085 trace_id=51 func=iprope_fwd_auth_check line=688 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1"
id=20085 trace_id=51 func=br_fw_forward_handler line=537 msg="Allowed by Policy-1:"
id=20085 trace_id=51 func=__if_queue_push_xmit line=365 msg="send out via dev-VxLan-IPsec, dst-mac-02:09:0f:00:06:03"
id=20085 trace_id=51 func=ipsecdev_hard_start_xmit_vxlan line=185 msg="enter IPsec interface-VxLan-IPsec"
id=20085 trace_id=51 func=esp_output4 line=859 msg="IPsec encrypt/auth"
id=20085 trace_id=51 func=ipsec_output_finish line=498 msg="send to 198.51.100.253 via intf-port1"
 

## ICMP echo-reply from PC2 to PC1

 

id=20085 trace_id=52 func=print_pkt_detail line=4793 msg="vd-root received a packet(proto=1, 192.168.100.2:2871->192.168.100.1:0) from VxLan-IPsec. type=0, code=0, id=2871, seq=1."
id=20085 trace_id=52 func=resolve_ip_tuple_fast line=4857 msg="Find an existing session, id-00002434, reply direction"
id=20085 trace_id=52 func=br_ipv4_fast_cb line=68 msg="enter fast path"
id=20085 trace_id=52 func=__if_queue_push_xmit line=365 msg="send out via dev-port2, dst-mac-02:09:0f:00:02:02"
 
 
 
Verify the session
 
FG1 # diag sys session filter clear
 
FG1 # diag sys session filter dst 192.168.100.2
 
FG1 # diag sys session filter proto 1
 
FG1 # diag sys session list
 
session info: proto=1 proto_state=00 duration=8 expire=51 timeout=0 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
ha_id=0 policy_dir=0 tunnel=VxLan-IPsec/ vlan_cos=0/255
state=may_dirty br
statistic(bytes/packets/allow_err): org=84/1/1 reply=84/1/1 tuples=2
tx speed(Bps/kbps): 10/0 rx speed(Bps/kbps): 10/0
orgin->sink: org pre->post, reply pre->post dev=4->21/21->4 gwy=0.0.0.0/0.0.0.0
hook=pre dir=org act=noop 192.168.100.1:2871->192.168.100.2:8(0.0.0.0:0)
hook=post dir=reply act=noop 192.168.100.2:2871->192.168.100.1:0(0.0.0.0:0)
misc=0 policy_id=1 auth_info=0 chk_client_info=0 vd=0
serial=00002434 tos=ff/ff app_list=0 app=0 url_cat=0
dd_type=0 dd_mode=0
 
total session 1
 
 
Example of a decrypted VxLAN-IPsec packet containing PC1’s Ethernet frame
 

 

## The ESP (IPsec) packet
 
Ethernet II, Src: MS-NLB-PhysServer-09_0f:00:04:02 (02:09:0f:00:04:02), Dst: MS-NLB-PhysServer-09_0f:00:04:01 (02:09:0f:00:04:01)
    Destination: MS-NLB-PhysServer-09_0f:00:04:01 (02:09:0f:00:04:01)
    Source: MS-NLB-PhysServer-09_0f:00:04:02 (02:09:0f:00:04:02)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 198.51.100.1, Dst: 203.0.113.2

 

    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 184
    Identification: 0xc28d (49805)
    Flags: 0x00
    Fragment offset: 0
    Time to live: 63
    Protocol: Encap Security Payload (50)
    Header checksum: 0x524f [correct]
    Source: 198.51.100.1
    Destination: 203.0.113.2
Encapsulating Security Payload
    ESP SPI: 0xa8d52b6d (2832542573)
    ESP Sequence: 12
    ESP IV: 5b8204e860038607
    ESP Pad Length: 0
    Next header: IPIP (0x04)
    Authentication Data [correct]

## The VxLAN packet carried inside the ESP (IPsec) packet

 

 
Internet Protocol Version 4, Src: 198.51.100.1, Dst: 203.0.113.2

 

    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 134
    Identification: 0xc28b (49803)
    Flags: 0x00
    Fragment offset: 0
    Time to live: 63
    Protocol: UDP (17)
    Header checksum: 0x52a4 [correct]
    Source: 198.51.100.1
    Destination: 203.0.113.2
User Datagram Protocol, Src Port: 4789, Dst Port: 4789

 

    Source Port: 4789
    Destination Port: 4789
    Length: 114
Virtual eXtensible Local Area Network
    Flags: 0x0000
        0... .... .... .... = GBP Extension: Not defined
        .... .... .0.. .... = Don't Learn: False
        .... 0... .... .... = VXLAN Network ID (VNI): False
        .... .... .... 0... = Policy Applied: False
        .000 .000 0.00 .000 = Reserved(R): 0x0000
    Group Policy ID: 0
    VXLAN Network Identifier (VNI): 0
    Reserved: 0
 
## PC1’s frame carried inside the VxLAN packet
 
Ethernet II, Src: MS-NLB-PhysServer-09_0f:00:02:02 (02:09:0f:00:02:02), Dst: MS-NLB-PhysServer-09_0f:00:06:03 (02:09:0f:00:06:03)

 

    Destination: MS-NLB-PhysServer-09_0f:00:06:03 (02:09:0f:00:06:03)
    Source: MS-NLB-PhysServer-09_0f:00:02:02 (02:09:0f:00:02:02)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.168.100.1, Dst: 192.168.100.2

 

    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 84
    Identification: 0xcfff (53247)
    Flags: 0x02 (Don't Fragment)
    Fragment offset: 0
    Time to live: 64
    Protocol: ICMP (1)
    Header checksum: 0x2155 [correct]
    Source: 192.168.100.1
    Destination: 192.168.100.2
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0
    Checksum: 0xc1c9 [correct]
    [Checksum Status: Good]
    Identifier (BE): 2888 (0x0b48)
    Identifier (LE): 18443 (0x480b)
    Sequence number (BE): 1 (0x0001)
    Sequence number (LE): 256 (0x0100)
    Data (48 bytes)



 

Labels:
67429
Submit Article Idea
Contributors
Broad. Integrated. Automated.

The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.

Social Media
Security Research
Company
News & Articles
Contact Us

Copyright 2024 Fortinet, Inc. All Rights Reserved.