| Description | This article describes how to identify and fix flash memory exhaustion issues on 30D, 30E and 50E clusters. |
| Scope | FortiOS 6.2.6 to 6.2.10. |
| Solution | Due to the increasing number and size of FortiGuard Databases, some low-end devices, namely FGT30D, FGT30D rugged, FGT50E, and FGT51E, could run into flash memory exhaustion. This could lead to several unwanted issues, namely:
- Partial or total configuration loss in case of a power outage or hard reboot. - FortiGuard databases or IPS engine upgrade failures. - Firmware upgrade failures.
To address this issue, since FortiOS 6.2.11 GA, some database size adjustments have been made, to reduce flash memory occupancy and avoid unexpected scenarios.
Thus, to fix this issue, it is strongly recommended to upgrade the cluster to 6.2.12 GA, keeping in mind that 6.2.11 is vulnerable to SSL VPN Buffer-Heap vulnerability (see https://www.fortiguard.com/psirt/FG-IR-22-398). Before performing the upgrade, freeing a portion of flash memory is mandatory to complete the upgrade successfully.
NOTE: Since FortiGuard databases are synced between primary and secondary nodes, it is very important to follow the flash cleanup and upgrade operations very carefully.
*******Issue Identification*******
If the cluster is experiencing one of the issues reported, please check the output of these CLI commands:
# fnsysctl df -h
Filesystem Size Used Available Use% Mounted on rootfs 1011.9M 68.6M 943.3M 7% / tmpfs 1011.9M 68.6M 943.3M 7% / none 1.6G 118.2M 1.4G 7% /tmp none 1.6G 468.0K 1.6G 0% /dev/shm none 1.6G 23.0M 1.5G 1% /dev/cmdb /dev/mtd6 18.0M 17.7M 244.0K 99% /data -----> flash partition. /dev/mtd7 30.0M 15.9M 14.0M 53% /data2
Or, alternatively,
# diagnose sys flash list
Partition Image TotalSize(KB) Used(KB) Use% Active 1 FGT50E-6.00-FW-build0272-190716 18432 14668 80% No 2 FGT50E-6.02-FW-build1142-200819 18432 18188 99% Yes 3 ETDB-1.00000 30720 16372 53% No
If the Usage value is in the range of 98%-99%, flash memory exhaustion is in place.
Check also the output of:
# fnsysctl ls -la /data/etc
<output_omitted> ----rw-rw- 1 0 0 Tue Feb 8 01:54:34 2022 2956003 geoip_db.gz
This command will be useful for the upgrade procedure
******Upgrade Procedure to fix flash memory issue*******
1) On Primary FortiGate, set an HA priority higher than the one of the Secondary Unit (default is 100), and only then enable HA override on the cluster.
- Primary (example):
# conf system ha set priority 200 end
- Secondary (example):
# conf system ha set priority 100 end
After priority configuration, on both nodes (primary first):
# conf system ha set override enable end
2) Temporarily disable the scheduled FortiGuard updates from System -> FortiGuard.
3) On the primary device, run the command: diagnose geoip delete-geoip-db. This command will delete the FortiGuard GeoIP Database (geoip_db.gz), freeing about 15% of memory without traffic impact.
Note: The device will be forced to reboot but, due to overriding, will be elected as primary again. This will prevent the secondary to become primary and resynchronize the GeoIP Database.
After the reboot, check the output of fnsysctl ls -la /data/etc to verify that the entry geoip_db.gz has been deleted.
Then, check the output of diagnose sys flash list, the used space for the active partition should have decreased to 85-86%. Check also the HA sync status with get sys ha status: the cluster will be displayed in-sync, even if the secondary has still the GeoIP Database stored in memory:
HA Health Status: OK Model: FortiGate-50E Mode: HA A-P Group: 146 Debug: 0 Cluster Uptime: 0 days 21:42:53 Cluster state change time: 2022-03-12 11:40:51
<output_omitted>
FGT50E3U15002795 (updated 5 seconds ago): in-sync FFGT50E3U15000151 (updated 4 seconds ago): in-sync
4) If all the previous checks were successful, issue diagnose geoip delete-geoip-db on the secondary unit and, after reboot, perform all the previous memory and HA sync checks to ensure that also the secondary unit has enough memory to perform the upgrade and that the cluster is fully operational.
5) Perform the uninterruptible upgrade following the recommended upgrade path. Remember to check the free space before each upgrade step.
6) Once the upgrade is completed, re-enable the Scheduled FortiGuard updates and run the command execute update-now on Primary Unit to resync all the Databases.
7) At the end of the procedure, the used memory shown by diagnose sys flash list will be in the range of 89% - 92%. This is an acceptable value that will prevent the cluster from configuration loss or upgrade failures. |
