DescriptionMaximum number of documents in an Elasticsearch shard
Installation and Administration
SolutionElasticsearch has a maximum limit on the number of documents (entries) that can be stored in a single shard, this limit is 2^31 (2147483519).
ZoneFox stores all its data in an Elasticsearch database, including system and user events, and for large deployments of agents it is possible that this maximum limit can be reached when there is an insufficient number of Elasticsearch shards available for storage. If the limit is reached, the Elasticsearch log file will contain entries like:
[2017-06-27 12:47:47,829][DEBUG][action.bulk ] [zfx-data01] [events.usr.2017.06][0] failed to execute bulk item (index) index {[events.usr.2017.06]
[evs][Gh75Nhdkiit90hXMighQRm], source[{"id":"Gh75Nhdkiit90hXMighQRm","m":"pc1","u":"zfx_admin01","app":"google chrome","act":"file
written","r":"/users/zfx_admin01/library/application support/google/chrome/default/local storage/https_example.com_0.localstorage"}]}
java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519
No further data will be written to the index in the database, and the log file will grow rapidly in size as these errors are repeated.
A workaround is to close the problematic index, preventing further write attempts to it.
A longer term solution is to add more shards for each index to increase the capacity. The typical Elasticsearch installation has one shard per node for each index, options include adding more Elasticsearch nodes (i.e have more Linux servers) or configure the existing Elasticsearch database to double the number of shards per node (but note this would increase the load on each Linux server).
