<p dir="auto"><strong>Elasticsearch version</strong>: 2.3.5</p>
<p dir="auto"><strong>Plugins installed</strong>: [head, kopf, cloud_aws, marvel, license]</p>
<p dir="auto"><strong>JVM version</strong>: 1.8.0_51</p>
<p dir="auto"><strong>OS version</strong>: CentOS 6.6</p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>:</p>
<p dir="auto"><strong>Steps to reproduce</strong>:</p>
<ol dir="auto">
<li>Cluster has 21 nodes. Previous version (1.7.2) worked with no issues for more than a year.</li>
</ol>
<p dir="auto">What we see is huge spike in write IOPS per node (500 writes/s before 2500 writes/s now). Around noon we have a script that reads a lot of data from this cluster and our read IOPS go high as well. We are running on AWS and the volume has only 3,000 max. With all those IOPS we hit the limit of 3,000 and at this point Elastisearch goes down.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/5177657/18093612/16ea9248-6e96-11e6-9fcc-9acda254116b.png"><img src="https://cloud.githubusercontent.com/assets/5177657/18093612/16ea9248-6e96-11e6-9fcc-9acda254116b.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">When we look at all ES nodes and get the status of Elasticsearch service we get this: <code class="notranslate">elasticsearch dead but subsys locked</code>. When we try to start ES backup we get this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
Starting elasticsearch: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000429990000, 15408234496, 0) failed; error='Cannot allocate memory' (errno=12)

 There is insufficient memory for the Java Runtime Environment to continue.
 Native memory allocation (mmap) failed to map 15408234496 bytes for committing reserved memory.
 An error report file with more information is saved as:
 /tmp/hs_err_pid8710.log
[FAILED]"><pre class="notranslate"><code class="notranslate">
Starting elasticsearch: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000429990000, 15408234496, 0) failed; error='Cannot allocate memory' (errno=12)

 There is insufficient memory for the Java Runtime Environment to continue.
 Native memory allocation (mmap) failed to map 15408234496 bytes for committing reserved memory.
 An error report file with more information is saved as:
 /tmp/hs_err_pid8710.log
[FAILED]
</code></pre></div>
<p dir="auto">So apparently init script does not correctly report Elasticsearch status - ES is still running but it is in a bad state. So we have to kill ES manually with kill -9 (which is not fun at all on 21 servers) and only then we can start ES back online.</p>
<p dir="auto">So with that multiple questions:</p>
<ol dir="auto">
<li>Why does new ES has so much more write IOPS? How do we prevent this?</li>
</ol>
<p dir="auto">One possibility is that auto-throttling is in place? We used to have this setting in our previous cluster:</p>
<p dir="auto">indices.store.throttle.max_bytes_per_sec = "100mb"</p>
<p dir="auto">So 2.3.5 apparently has auto-throttling enabled by default. Since it is impossible to delete our previous setting - does new ES just ignores it? Or how do we delete it? How can we disable auto-throttling if it causes the issue?</p>
<ol dir="auto">
<li>What is wrong with init script? We use official rpm and it seems there is a bug with init script - it does not correctly detect hung process. How can we fix that?</li>
</ol>
<p dir="auto">Thanks a lot!</p>
<p dir="auto">Hi,</p>
<p dir="auto">looks like <code class="notranslate">DocumentMapper</code> is creating its own list of root field mappers:</p>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=" ...
            // UID first so it will be the first stored field to load (so will benefit from &quot;fields: []&quot; early termination
            this.rootMappers.put(UidFieldMapper.class, new UidFieldMapper());
            this.rootMappers.put(IdFieldMapper.class, idFieldMapper);
            this.rootMappers.put(RoutingFieldMapper.class, new RoutingFieldMapper());
            // add default mappers, order is important (for example analyzer should come before the rest to set context.analyzer)
            this.rootMappers.put(SizeFieldMapper.class, new SizeFieldMapper());
            this.rootMappers.put(IndexFieldMapper.class, new IndexFieldMapper());
            this.rootMappers.put(SourceFieldMapper.class, new SourceFieldMapper());
            this.rootMappers.put(TypeFieldMapper.class, new TypeFieldMapper());
            this.rootMappers.put(AnalyzerMapper.class, new AnalyzerMapper());
            this.rootMappers.put(AllFieldMapper.class, new AllFieldMapper());
            this.rootMappers.put(BoostFieldMapper.class, new BoostFieldMapper());
            this.rootMappers.put(TimestampFieldMapper.class, new TimestampFieldMapper());
            this.rootMappers.put(TTLFieldMapper.class, new TTLFieldMapper());
            this.rootMappers.put(ParentFieldMapper.class, new ParentFieldMapper());
 ..."><pre class="notranslate"> ...
            <span class="pl-c">// UID first so it will be the first stored field to load (so will benefit from "fields: []" early termination</span>
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">UidFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">UidFieldMapper</span>());
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">IdFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-s1">idFieldMapper</span>);
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">RoutingFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">RoutingFieldMapper</span>());
            <span class="pl-c">// add default mappers, order is important (for example analyzer should come before the rest to set context.analyzer)</span>
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">SizeFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">SizeFieldMapper</span>());
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">IndexFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">IndexFieldMapper</span>());
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">SourceFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">SourceFieldMapper</span>());
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">TypeFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">TypeFieldMapper</span>());
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">AnalyzerMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">AnalyzerMapper</span>());
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">AllFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">AllFieldMapper</span>());
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">BoostFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">BoostFieldMapper</span>());
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">TimestampFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">TimestampFieldMapper</span>());
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">TTLFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">TTLFieldMapper</span>());
            <span class="pl-smi">this</span>.<span class="pl-s1">rootMappers</span>.<span class="pl-en">put</span>(<span class="pl-smi">ParentFieldMapper</span>.<span class="pl-k">class</span>, <span class="pl-k">new</span> <span class="pl-smi">ParentFieldMapper</span>());
 ...</pre></div>
<p dir="auto">and <code class="notranslate">DocumentMapperParser</code> is creating its own:</p>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="...
        rootTypeParsers = new MapBuilder&lt;String, Mapper.TypeParser&gt;()
                .put(SizeFieldMapper.NAME, new SizeFieldMapper.TypeParser())
                .put(IndexFieldMapper.NAME, new IndexFieldMapper.TypeParser())
                .put(SourceFieldMapper.NAME, new SourceFieldMapper.TypeParser())
                .put(TypeFieldMapper.NAME, new TypeFieldMapper.TypeParser())
                .put(AllFieldMapper.NAME, new AllFieldMapper.TypeParser())
                .put(AnalyzerMapper.NAME, new AnalyzerMapper.TypeParser())
                .put(BoostFieldMapper.NAME, new BoostFieldMapper.TypeParser())
                .put(ParentFieldMapper.NAME, new ParentFieldMapper.TypeParser())
                .put(RoutingFieldMapper.NAME, new RoutingFieldMapper.TypeParser())
                .put(TimestampFieldMapper.NAME, new TimestampFieldMapper.TypeParser())
                .put(TTLFieldMapper.NAME, new TTLFieldMapper.TypeParser())
                .put(UidFieldMapper.NAME, new UidFieldMapper.TypeParser())
                .put(IdFieldMapper.NAME, new IdFieldMapper.TypeParser())
                .immutableMap();
..."><pre class="notranslate">...
        <span class="pl-s1">rootTypeParsers</span> = <span class="pl-k">new</span> <span class="pl-smi">MapBuilder</span>&lt;<span class="pl-smi">String</span>, <span class="pl-smi">Mapper</span>.<span class="pl-smi">TypeParser</span>&gt;()
                .<span class="pl-en">put</span>(<span class="pl-smi">SizeFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">SizeFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">IndexFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">IndexFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">SourceFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">SourceFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">TypeFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">TypeFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">AllFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">AllFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">AnalyzerMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">AnalyzerMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">BoostFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">BoostFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">ParentFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">ParentFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">RoutingFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">RoutingFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">TimestampFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">TimestampFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">TTLFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">TTLFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">UidFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">UidFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">put</span>(<span class="pl-smi">IdFieldMapper</span>.<span class="pl-c1">NAME</span>, <span class="pl-k">new</span> <span class="pl-smi">IdFieldMapper</span>.<span class="pl-smi">TypeParser</span>())
                .<span class="pl-en">immutableMap</span>();
...</pre></div>
<p dir="auto">From external plugin it is possible to modify only root type parsers from <code class="notranslate">DocumentMapperParser</code>, but it is the list from <code class="notranslate">DocumentMapper</code> that is used when creating <code class="notranslate">ParsedDocument</code> instance.</p>
<p dir="auto">BTW,<br>
im trying to create plugin that will add support for computed fields (based on script), i wanted to borrow some 'inspiration' from how '_all' field is implemented (as it is in a way computed field). Is implementing root field mapper the right way to do it ? I tried to use field mappers (not root), but looks like they are not used unless field value is present in indexed document (which is kinda problem for computed fields, as they are never present in source document)</p>