Technical Tip: HA out of sync after upgraded in multi VDOMs environment

ydong01 · ‎06-18-2019

Description

This article describes how to fix HA out of sync after upgrade in multi VDOMs environment.

When upgrade a cluster with multi VDOMs and managed by FortiManager, sometimes it will be out of sync as SSL-SSH profile checksum is different. By removing SSL exempt from SSL-SSH profile, it will make the cluster sync again.

Solution

Troubleshooting:

The cluster is upgraded smoothly, but slave unit is out-of-sync. Verify if SSL-SSH profile "custom-deep-inspection" is different. Run the following commands:

1)

get sys ha status

FGyyy (updated 0 seconds ago): out-of-sync
FGxxx(updated 2 seconds ago): in-sync

2)

dia sys ha checksum cluster

FGxxx
global: a0 f4 40 8d 35 29 b3 f8 7b 88 cd be ee 70 f9 a6
root: 37 98 9e 8a a6 52 c4 05 39 e5 3c 7c bd 16 62 b2
…
all: 63 20 fd 0a d9 27 07 f2 2a df ce 92 6e 83 4f 87

FGyyy
global: a0 f4 40 8d 35 29 b3 f8 7b 88 cd be ee 70 f9 a6
root: d4 3d 0b e2 f5 73 eb 7e 56 44 00 14 2d 3d 99 7a
…
all: 4e 3a ba 57 6e d6 34 25 53 ac 49 e9 bf b0 64 9c

3)

dia sys ha checksum show root

FGxxx
…
firewall.ssl-ssh-profile: b6205ab462401c94041894406b35b576
…

FGyyy
…
firewall.ssl-ssh-profile: f5277eb4bb806414c11216aa303140e5
…

4)

dia sys ha checksum show root firewall.ssl-ssh-profile

FGxxx
certificate-inspection: fa03bfe69adbe3f18e67a79a630e8e8f
custom-deep-inspection: a40395f9be9b3406e6f05c5712b3ebfb
deep-inspection: a1a2588c7b0df7715dcc674a2353f3f9

FGyyy
certificate-inspection: fa03bfe69adbe3f18e67a79a630e8e8f
custom-deep-inspection: 7f7d6b8625b969a31d21f12402a2f9fc
deep-inspection: a1a2588c7b0df7715dcc674a2353f3f9

5)

show firewall ssl-ssh-profile

FGxxx
config firewall ssl-ssh-profile
    edit "custom-deep-inspection"
      …
        config ssl-exempt
            …
            edit 3
                set type address
                set address "google-play"
            next
            edit 4
                set type address
                set address "update.microsoft.com"
            next
            edit 5
                set type address
                set address "swscan.apple.com"
            next
            edit 6
                set type address
                set address "autoupdate.opera.com"
            next
            edit 7
                set type wildcard-fqdn
            …
            edit 30
                set type wildcard-fqdn
            next
end

FGyyy
config firewall ssl-ssh-profile
    edit "custom-deep-inspection"
        …
        config ssl-exempt
            …
            edit 3
                set type address
                set address "google-play"
            next
            edit 4
                set type address
                set address "update.microsoft.com"
            next
            edit 5
                set type address
                set address "swscan.apple.com"
            next
            edit 6
                set type address
                set address "autoupdate.opera.com"
            next
end

Analysis:

1) Check customer backup configuration file before upgrading, there was no custom-deep-inspection profile.

2) Customer use FortiManager manage the cluster, in FortiManager, under same ADOM, the other cluster use custom-deep-inspection profile.

3) Customer may use custom-deep-inspection profile in the future for this upgrade cluster.

4) Master unit contain 30 exempt and 4 of them not resolve as no FQDN address, Slave unit contain 4 not resolved FQDN address in exempt, so it cause cluster out-of-sync.

Solution:

1) Upgrade to 6.0.5 should fix it.

2) If could not upgrade, remove 4 FQDN Address from “Exempt from SSL Inspection“ under “customer-deep-inspection” from master unit also can make cluster sync again.