<p dir="auto">Hi,<br>
Elasticsearch currently doesn't have a parameter which switches indices in append only mode. This could be interesting from performance point of view.</p>
<p dir="auto">We are dealing with a problem of limited ingestion rate in ES. We tried to focus on a couple of hot spots to resolve potential bottlenecks. One of them was DocValues merges for sparse fields, then after changing approach synchronous translog became a bottleneck. It even seems that contention problem is not negligible: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="151747455" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/18053" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/18053/hovercard" href="https://github.com/elastic/elasticsearch/issues/18053">#18053</a>. These are only a few places where we tried to find optimizations. While analyzing profiles in Java Mission Control it was hard to ignore the another major problem: versions lookup. See the image below.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/6692291/14913686/309efd74-0e05-11e6-9437-f0497a9bcaca.png"><img src="https://cloud.githubusercontent.com/assets/6692291/14913686/309efd74-0e05-11e6-9437-f0497a9bcaca.png" alt="selection_383" style="max-width: 100%;"></a></p>
<p dir="auto">Profiler shows that versions lookups are taking up to 25% of CPU time. This is a lot. Of course one could say I am showing a very specific case because I am using tiny documents in tests. I think this is not exactly true. Elasticsearch stack is much focused on logs aggregation and recently also metrics (Logstash, Kibana, Marvel, Beats, ...). Most systems designed for metric or log aggregation indices are append-only. Of course ES is general purpose distributed search and analytics engine so document versions are really important ...  but in many cases append-only mode could be sufficient.<br>
One of the major strengths of Lucene internally is that segments are append only. Why not to use this simple but powerful assumption on higher level? Not all ES use cases require concurrent update functionality in which versions control really matters. What if Elasticsearch indices are marked as append-only (on they are created. Indices have properties which cannot be changed after index is created (eg. number of shards). This could be another parameter which can be applied only on index creation.</p>
<p dir="auto">The contention problem mentioned in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="151747455" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/18053" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/18053/hovercard" href="https://github.com/elastic/elasticsearch/issues/18053">#18053</a> would also become less painful in append-only indices. Version lookup is inside synchronized block in InternalEngine#innerIndex. This means threads are in synchronized section for a longer time than needed (in case where versions are not needed).</p>
<p dir="auto">During tests we disabled versions control in Elasticsearch which brought us significant improvement in indexing speed. Look at the chart below. Indexing speed on the same indices (without removing old data - just changed ES version) increased by 50% only by removing a version control.</p>
<p dir="auto">My test environment is not very powerful. Only 2 machines with 4 CPU cores each. Indices have only one replica.</p>
<p dir="auto"><a href="https://apps.sematext.com/spm-reports/s/ucSD8cuTRr" rel="nofollow">https://apps.sematext.com/spm-reports/s/ucSD8cuTRr</a><br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/6692291/14913721/59e14ad4-0e05-11e6-8a7b-ab88f398e625.png"><img src="https://cloud.githubusercontent.com/assets/6692291/14913721/59e14ad4-0e05-11e6-8a7b-ab88f398e625.png" alt="selection_397" style="max-width: 100%;"></a></p>
<p dir="auto">I would really appreciate any feedback. Am I missing something obvious? Thanks</p>
<p dir="auto">Hello,</p>
<p dir="auto">Right now if you enable Elasticsearch slow log from config - elasticsearch.yml or through api call, it will generate the logs on the data node for queries that take more then 1 second (if set like this) in ES_index_search_slowlog.log.<br>
The problem is that it will generate these logs on the data nodes where the query hits, so you have to check each nodes to see them If you have 50 data nodes this can be quite painfully.<br>
What we need is for the these logs to be generated at the node the request is sent to ( client node for example)  and the logs should also include the ip of the machine the call was made from.</p>
<p dir="auto">I believe this should be implemented as it will make it much easier to debug.</p>
<p dir="auto">**<em>I know you could manually collect all the logs and use nginx proxy in front to catch the ip and then concatenate all of them.  But why do all of this ? ***</em></p>