<p dir="auto"><strong>Elasticsearch version</strong>: 2.3.3</p>
<p dir="auto"><strong>JVM version</strong>: OpenJDK 1.8.0_91</p>
<p dir="auto"><strong>OS version</strong>: Ubuntu 16.04</p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>:<br>
Documents are not properly indexed when Elasticsearch is restarted just after bulk request is send.<br>
Bulk request is retried because of <em>connection refused</em> error. Finally random documents are duplicated (same content, different id) in the index.</p>
<p dir="auto"><strong>Steps to reproduce</strong>:</p>
<ol dir="auto">
<li>
<p dir="auto">Start sending 1000 bulk requests with 100 documents each, one after another. First request example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XPOST 'localhost:9200/_bulk' -d '
           {&quot;index&quot;:{&quot;_index&quot; : &quot;index1&quot;, &quot;_type&quot;: &quot;test&quot;}}
           {&quot;counter&quot;: &quot;1&quot;, &quot;package&quot;: &quot;1/1000&quot;}
           {&quot;index&quot;:{&quot;_index&quot; : &quot;index1&quot;, &quot;_type&quot;: &quot;test&quot;}}
           {&quot;counter&quot;: &quot;2&quot;, &quot;package&quot;: &quot;1/1000&quot;}
           ...
           {&quot;index&quot;:{&quot;_index&quot; : &quot;index1&quot;, &quot;_type&quot;: &quot;test&quot;}}
           {&quot;counter&quot;: &quot;100&quot;, &quot;package&quot;: &quot;1/1000&quot;}
'"><pre class="notranslate"><code class="notranslate">curl -XPOST 'localhost:9200/_bulk' -d '
           {"index":{"_index" : "index1", "_type": "test"}}
           {"counter": "1", "package": "1/1000"}
           {"index":{"_index" : "index1", "_type": "test"}}
           {"counter": "2", "package": "1/1000"}
           ...
           {"index":{"_index" : "index1", "_type": "test"}}
           {"counter": "100", "package": "1/1000"}
'
</code></pre></div>
</li>
<li>
<p dir="auto">Restart Elasticsearch just after server starts receiving requests.<br>
<code class="notranslate">sudo service elasticsearch restart</code></p>
</li>
<li>
<p dir="auto">Repeat previous step couple of times.</p>
</li>
<li>
<p dir="auto">If bulk request fails (Connection refused) because of Elasticsearch restart, repeat sending request until it succeed.</p>
</li>
<li>
<p dir="auto">After all bulk requests are sent, index document count is grater then 10000.<br>
<code class="notranslate">curl -XGET http://localhost:9200/index1/_count</code><br>
{"count":<strong>10326</strong>,"_shards":{"total":5,"successful":5,"failed":0}}</p>
</li>
</ol>
<p dir="auto"><strong>Provide logs (if relevant)</strong>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2016-07-06 15:49:28,441][INFO ][node                     ] [ubuntu1-node] stopping ...
[2016-07-06 15:49:28,485][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
    at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
    at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
    at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
    at org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
    at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
    at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
    at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
    at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:146)
    at org.elasticsearch.rest.action.support.RestResponseListener.processResponse(RestResponseListener.java:43)
    at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49)
    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89)
    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85)
    at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:356)
    at org.elasticsearch.action.bulk.TransportBulkAction$2.onFailure(TransportBulkAction.java:351)
    at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:567)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:527)
    at org.elasticsearch.transport.TransportService$2.run(TransportService.java:206)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2016-07-06 15:49:28,512][WARN ][transport                ] [ubuntu1-node] Transport response handler not found of id [43]
[2016-07-06 15:49:28,863][INFO ][node                     ] [ubuntu1-node] stopped
[2016-07-06 15:49:28,863][INFO ][node                     ] [ubuntu1-node] closing ...
[2016-07-06 15:49:28,868][INFO ][node                     ] [ubuntu1-node] closed"><pre class="notranslate"><code class="notranslate">[2016-07-06 15:49:28,441][INFO ][node                     ] [ubuntu1-node] stopping ...
[2016-07-06 15:49:28,485][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
    at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
    at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
    at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
    at org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
    at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
    at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
    at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
    at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:146)
    at org.elasticsearch.rest.action.support.RestResponseListener.processResponse(RestResponseListener.java:43)
    at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49)
    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89)
    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85)
    at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:356)
    at org.elasticsearch.action.bulk.TransportBulkAction$2.onFailure(TransportBulkAction.java:351)
    at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:567)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:527)
    at org.elasticsearch.transport.TransportService$2.run(TransportService.java:206)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2016-07-06 15:49:28,512][WARN ][transport                ] [ubuntu1-node] Transport response handler not found of id [43]
[2016-07-06 15:49:28,863][INFO ][node                     ] [ubuntu1-node] stopped
[2016-07-06 15:49:28,863][INFO ][node                     ] [ubuntu1-node] closing ...
[2016-07-06 15:49:28,868][INFO ][node                     ] [ubuntu1-node] closed
</code></pre></div>
<p dir="auto">From Binh:</p>
<p dir="auto">“Thank you for reporting this, the plan is to revert back and allow 0 for external versioning. Not sure yet when it will be but likely in the next 1 or 2 releases.”</p>
<p dir="auto">---------- Forwarded message ----------<br>
Hello team - I just switched over to 1.1.0 and changed from VersionType.EXTERNAL to VersionType.EXTERNAL_GTE. Ever since then, we are seeing the error below.</p>
<p dir="auto">Any comments?</p>
<p dir="auto">I looked at ES code and see this newly added line for EXTERNAL_GTE. Does it mean we dont support zero as a valid version number? Our systems use zero as the start version number.</p>
<p dir="auto">Also to add: it seems that even VersionType.EXTERNAL is also broken in the new version (1.1.0) since it now does the validation of input version number and it does not allow a save if the input version number is zero.<br>
Looking for inputs from ES team on how to proceed on this.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="       @Override
        public boolean validateVersion(long version) {
            return version &gt; 0L;
        }

org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: illegal version value [0] for version type [EXTERNAL_GTE];
    at org.elasticsearch.action.ValidateActions.addValidationError(ValidateActions.java:29)
    at org.elasticsearch.action.index.IndexRequest.validate(IndexRequest.java:181)
    at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:63)
    at org.elasticsearch.client.transport.support.InternalTransportClient$2.doWithNode(InternalTransportClient.java:109)
    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:228)
    at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
    at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:82)
    at org.elasticsearch.client.transport.TransportClient.index(TransportClient.java:330)
    at org.elasticsearch.action.index.IndexRequestBuilder.doExecute(IndexRequestBuilder.java:314)
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85)
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)"><pre class="notranslate"><code class="notranslate">       @Override
        public boolean validateVersion(long version) {
            return version &gt; 0L;
        }

org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: illegal version value [0] for version type [EXTERNAL_GTE];
    at org.elasticsearch.action.ValidateActions.addValidationError(ValidateActions.java:29)
    at org.elasticsearch.action.index.IndexRequest.validate(IndexRequest.java:181)
    at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:63)
    at org.elasticsearch.client.transport.support.InternalTransportClient$2.doWithNode(InternalTransportClient.java:109)
    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:228)
    at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
    at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:82)
    at org.elasticsearch.client.transport.TransportClient.index(TransportClient.java:330)
    at org.elasticsearch.action.index.IndexRequestBuilder.doExecute(IndexRequestBuilder.java:314)
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85)
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)
</code></pre></div>
<p dir="auto">-Amit</p>