DescriptionIn case of customer having older FortiSOAR (CyOPs) community editions the elasticsearch logs fill up rapidly due to the generation of the license expired issue every second. Due to this the /var/log partition disk space gets fills up, which might also cause the elasticsearch service to stop.
To check if this is the issue, you can run the following diagnostic checks:
- Check the disk space using the df –h command.
- Check the elasticsearch log file using the following command:
tailf /var/log/elasticsearch/elasticsearch.log
In the log you will see the following error (Block1 - Elasticsearch error) repeating every second:
SolutionTo resolve this issue, do the following:
- Enable the Basic License
- Remove Logstash and Kibana
- Delete unassigned shards
Enabling the basic license
- To stop the Block1 - Elasticsearch error, you need to enable the basic license using the following command:
curl
--user sa-admin:cybersponse -X POST "https://localhost:9200/_license/start_basic?acknowledge=true"
-H 'Content-Type: application/json'
Note: You need to use http or https (highlighted in RED) and sa-admin:cybersponse or elastic:<hkey> based on the version of your community edition.
You will see the following output:
{"acknowledged":true,"basic_was_started":true}
- Restart the elasticsearch service using the following command:
systemctl restart elasticsearch
Removing Logstash and Kibana
- To remove logstash, first you must stop the logstash service, then disable the logstash service, and then remove the logstash service, using the following commands:
systemctl stop logstash
systemctl disable logstash
yum remove logstash
- To remove kibana, first you must stop the kibana service, then disable the kibana service, and then remove the kibana service, using the following commands:
systemctl stop kibana
systemctl disable kibana
yum remove kibana
- Reboot your FortiSOAR (CyOPs) instance.
Deleting Unassigned Shards
- Check the elasticsearch.log for the following error:
/var/lib/elasticsearch/nodes/0/indices/9fksydJQSv-bKLExc6mv7g/4/_state: Too many open files
- Delete
unassigned shards indices using the following commands:
curl
--user sa-admin:cybersponse -X GET "https://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE --user sa-admin:cybersponse "https://localhost:9200/{}"
curl --user
sa-admin:cybersponse -X DELETE
"https://localhost:9200/.watcher-*?pretty"
Note: You need to use http or https (highlighted in RED) and sa-admin:cybersponse or elastic:<hkey> based on the version of your community edition.
- Restart all the services using the following command:
csadm services --restart