Optimal Shard Placement in a Petabyte Scale Elasticsearch Cluster
Interesting overview.<p>I understand from elsewhere on their site that they are not using a recent version of ES (2.x?). This has a big impact on how things behave as they improved performance and garbage collect behavior by e.g. doing more things off heap with memory mapped files, fixing tons of issues with clustering, refactoring the on disk storage in Lucene 5 and 6, and giving users more control over sharding and indicex creation. So, it&#x27;s likely that this setup is working around issues that probably have been addressed in later versions.<p>For example, 6.x has a shard allocation API that allows you control which nodes host which shards. That sounds like that would be useful in this setup.<p>They also seem to be mixing query and indexing traffic on the same nodes. I would suggest separating these in a cluster this big and using specialized  nodes for query and index nodes (as well as dedicated master nodes). That way write node performance would be a lot more predictable and query traffic is no longer able to disrupt writes. Also you can use instances with more CPU&#x2F;memory for querying or use elastic scaling groups to add querying capacity when needed. You might even get away with using cheaper T instances for this with cpu credits.