Technical Note: ZoneFox 3 - How to manually reallocate an unassigned shard in Elasticsearch

[FortiKoala]

Description

How to manually reallocate an unassigned shard in Elasticsearch

Scope

Installation and Administration

Solution

The ZoneFox System Status, which can be viewed in detail on the ADMINISTRATION - System Status page, will show when there is a problem with the underlying Elasticsearch database.  For example when there is an unassigned shard:

Note that this can be a temporary status, while Elasticsearch performs housekeeping tasks and shards are moved between nodes.  However, if the problem does not resolve itself any unassigned shards can be manually reallocated to a database node.

You can see which exact index/shard is affected in Elasticsearch using a number of tools, for example in the head plugin (available via a URL in the format http://<database_server>:9200/_plugin/head) the problem shard is coloured grey:

If you click on the 3 in this case you can see some additional information:

Alternatively, you can use the _cat/shards query, for example http://<database_server>:9200/_cat/shards which will give output like:

events.sys.2017.07.07             4 p STARTED      1508644 234.2mb 10.20.30.44 ZoneFox_Node_10.20.30.44 events.sys.2017.07.07             1 p STARTED      1540013 239.5mb 10.20.30.41 ZoneFox_Node_10.20.30.41  events.sys.2017.07.07             2 p STARTED      1517181 236.8mb 10.20.30.42 ZoneFox_Node_10.20.30.42 events.sys.2017.07.07             3 p UNASSIGNED                                                               events.sys.2017.07.07             0 p STARTED      1500873 236.2mb 10.20.30.40 ZoneFox_Node_10.20.30.40 

This can be fixed by manually reallocating the shard to a node.  While this can be done at the command line using curl, the following instructions show how to do this in a web browser using the Elasticsearch head plugin.

1. Connect to your Elasticsearch instance using the head plugin, e.g http://<database_server>:9200/_plugin/head
2. Change to the Any Request tab
3. Select POST from the dropdown list
4. Add _cluster/reroute to the database URL under the Query section, e.g http://<database_server>:9200/_cluster/reroute
5. Enter the required allocate command, and enter the correct details for the index name, shard number and node name to be allocated to.  Note that if the shard belongs to an events index you must also provide the allow_primary flag.  For example:
   

   {    "commands": [{        "allocate": {            "index": "<index_name>",            "shard": <shard_number>,            "node": "<node_name>",            "allow_primary": 1        }    }] }

   
   
6. Toggle on the Pretty view and click on the Validate JSON button before clicking on the Request button.  For example:
   
   
7. If you have multiple shards to reallocate, you can either specify each one individually and repeat the request, or you can give the command an array of allocate commands in a single request, each separated by a comma.