Troubleshooting Tip: Restoring full-config backup in Gen2 chassis device cause FPCs to go into dead state

[hvaradaraj]

Description	This article describes how restoring a Full-config backup in a Gen2 chassis device causes FPCs to go into a dead state and offers a solution.
Scope	Applicable for Gen2 FG-6300F, FG-6301F, FG-6500F and FG-6501F.
Solution	Use a config backup taken from the GUI or backup taken from the CLI using 'execute backup config'. In the 6K Gen2 device, the MBD has 32G memory, but FPC has 64G. Therefore, some FPC default configurations are unsuitable for MBD and vice versa. Full-config includes default hidden configurations which will cause FPCs to go into a dead state when a Full-config backup is restored. Findings: The default max size of memory global-setting is 337438883 on MBD, but it is 675833937 on FPC, the socket-size of IPS global default is 128 on MBD, and it is 256 on FPC as shown below. config log memory global-setting show full-configuration config log memory global-setting     set max-size 337438883        <-- MBD default max-size 337438883.     set full-first-warning-threshold 75     set full-second-warning-threshold 90     set full-final-warning-threshold 95 end config log memory global-setting show full-configuration config log memory global-setting     set max-size 675833937      <----- FPC default max-size 675833937.     set full-first-warning-threshold 75     set full-second-warning-threshold 90     set full-final-warning-threshold 95 end config ips global show full-configuration config ips global ......     set socket-size 128  <----- MBD default socket-size is 128. ...... end end config ips global show full-configuration config ips global ......     set socket-size 256    <----- FPC default socket-size is 256. .... end When a Full-config backup is restored, the MBD log memory max-size changes from '337438883'(default) to '675833937'(FPC default value) to trigger the problem as shown below. diagnose sys confsync showcsum global log.memory.global- ========================================================================== Slot: 1 Module SN: FPC6KFT0xxxxxx87 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 2 Module SN: FPC6KFT0xxxxxx86 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 3 Module SN: FPC6KFT0xxxxxx81 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 4 Module SN: FPC6KFT0xxxxxx38 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 5 Module SN: FPC6KFT0xxxxxx35 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 6 Module SN: FPC6KFT0xxxxxx08 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 7 Module SN: FPC6KFT0xxxxxx31 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 8 Module SN: FPC6KFT0xxxxxx00 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 9 Module SN: FPC6KFT0xxxxxx02 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 10 Module SN: FPC6KFT0xxxxxx09 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== MBD SN: F6KF51T0xxxxxx27 --- CSUM_TYPE_HA --- [max-size]='675833937': 2ebc2f834dfc75eb0aac40b3f55df930  --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- When a Full-config backup is restored, the MBD IPS socket max-size changes from '128' (default) to '256'(FPC default value) to trigger the problem. CH2-Gen02-27 (global) # diagnose sys confsync showcsum global ips.global ========================================================================== Slot: 1 Module SN: FPC6KFT0xxxxxx87 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 2 Module SN: FPC6KFT0xxxxxx86 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 3 Module SN: FPC6KFT0xxxxxx81 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 4 Module SN: FPC6KFT0xxxxxx38 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 5 Module SN: FPC6KFT0xxxxxx35 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 6 Module SN: FPC6KFT0xxxxxx08 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 7 Module SN: FPC6KFT0xxxxxx31 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 8 Module SN: FPC6KFT0xxxxxx00 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 9 Module SN: FPC6KFT0xxxxxx02 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== Slot: 10 Module SN: FPC6KFT0xxxxxx09 --- CSUM_TYPE_HA --- --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- ========================================================================== MBD SN: F6KF51T0xxxxxx27 --- CSUM_TYPE_HA --- [socket-size]='256': e8e51af4dcc754725b00e8e8906629c6  --- END --- --- CSUM_TYPE_CONFSYNC --- --- END --- As visible in the below output, all of the FPCs in the device went into a dead state after restoring the full-config backup. diagnose load-balance status ========================================================================== MBD SN: F6KF51T0xxxxxx27 Primary FPC Blade: N/A Slot 1: Status:Dead Function:Active Link: Base: Up Fabric: Up Heartbeat: Management: Good Data: Failed Status Message:"Waiting for configuration sync." Slot 2: Status:Dead Function:Active Link: Base: Up Fabric: Up Heartbeat: Management: Good Data: Failed Status Message:"Waiting for configuration sync." Slot 3: Status:Dead Function:Active Link: Base: Up Fabric: Up Heartbeat: Management: Good Data: Failed Status Message:"Waiting for configuration sync." Slot 4: Status:Dead Function:Active Link: Base: Up Fabric: Up Heartbeat: Management: Good Data: Failed Status Message:"Waiting for configuration sync." Slot 5: Status:Dead Function:Active Link: Base: Up Fabric: Up Heartbeat: Management: Good Data: Failed Status Message:"Waiting for configuration sync." Slot 6: Status:Dead Function:Active Link: Base: Up Fabric: Up Heartbeat: Management: Good Data: Failed Status Message:"Waiting for configuration sync." Slot 7: Status:Dead Function:Active Link: Base: Up Fabric: Up Heartbeat: Management: Good Data: Failed Status Message:"Waiting for configuration sync." Slot 8: Status:Dead Function:Active Link: Base: Up Fabric: Up Heartbeat: Management: Good Data: Failed Status Message:"Waiting for configuration sync." Slot 9: Status:Dead Function:Active Link: Base: Up Fabric: Up Heartbeat: Management: Good Data: Failed Status Message:"Waiting for configuration sync." Slot 10: Status:Dead Function:Active Link: Base: Up Fabric: Up Heartbeat: Management: Good Data: Failed Status Message:"Waiting for configuration sync." Solution: To get rid of the problem, restore the config backup taken from the GUI or the backup taken from the CLI using 'execute backup config'. Related article: Technical Tip: Information on FortiGate-6000F series Gen1 and Gen2.