<p dir="auto">I'm opening this to discuss possible options:</p>
<p dir="auto">I've been scrutinizing ES indexing performance on the NYC taxi data set (1.2 B taxi rides, numerics heavy: <a href="http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml" rel="nofollow">http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml</a>).</p>
<p dir="auto">These documents are small (24 fields, though a bit sparse with ~23% cells missing) and are almost entirely numbers (indexed as points + doc values).</p>
<p dir="auto">As a "ceiling" for indexing performance I also indexed the same data set using Lucene's "thin wrapper" demo server (<a href="https://github.com/mikemccand/luceneserver">http://github.com/mikemccand/luceneserver</a>), indexing the same documents as efficiently as I know how (see <code class="notranslate">indexTaxis.py</code>).</p>
<p dir="auto">The demo Lucene server has many differences vs. ES: it has no transaction log (does not periodically fsync), uses <code class="notranslate">addDocuments</code> not <code class="notranslate">updateDocument</code>, can index from a more compact documents source (190 GB CSV file, vs 512 GB json file for ES), does not add a costly <code class="notranslate">_uid</code> field (nor <code class="notranslate">_version</code>, <code class="notranslate">_type</code>) , uses a streaming bulk API, etc.  I disabled <code class="notranslate">_all</code> and <code class="notranslate">_source</code> in ES, but net/net ES is substantially slower than the demo Lucene server.</p>
<p dir="auto">So, one big thing I noticed that is maybe a lowish hanging fruit is that ES loses a lot of its indexing buffer to <code class="notranslate">LiveVersionMap</code>: if I give ES 1 GB indexing buffer, and index into only 1 shard, and disable refresh, the version map is taking ~2/3 of that buffer, leaving only ~1/3 for Lucene's <code class="notranslate">IndexWriter</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="node0: [2016-08-03 09:39:07,557][DEBUG][index.engine             ] [node0] [taxis][0] use refresh to write indexing buffer (heap size=[313.7mb]), to also clear version map (heap size=[730.3mb])"><pre class="notranslate"><code class="notranslate">node0: [2016-08-03 09:39:07,557][DEBUG][index.engine             ] [node0] [taxis][0] use refresh to write indexing buffer (heap size=[313.7mb]), to also clear version map (heap size=[730.3mb])
</code></pre></div>
<p dir="auto">This also means ES is necessarily doing periodic refresh when I didn't ask it to.</p>
<p dir="auto">This is quite frustrating because I don't need optimistic concurrency here, nor real-time gets, nor refreshes.  However, I fear the version map might be required during recovery, to ensure when playing back indexing operations from the transaction log that they do not incorrectly overwrite newer indexing operations?  But then, this use case is also append-only, so maybe during recovery we could safely skip that, if the user turns on this new setting.</p>
<p dir="auto">The version map makes an entry in a <code class="notranslate">HashMap</code> for each document indexed, and the entry stores non-trivial information, creating at least 4 new objects, holding longs/ints, etc.  If we can't make it turn-off-able maybe we should instead try to reduce its per-indexing-op overhead...</p>
<p dir="auto">the node level TTL Purger thread fires up bulk delete request that might trigger <code class="notranslate">auto_create_index</code> if the purger thread runs a bulk after the index has been deleted.</p>