<p dir="auto"><strong>Elasticsearch version</strong>: 2.3.2</p>
<p dir="auto">This is an attempt to simulate a bad disk that has turned read only.</p>
<ol dir="auto">
<li>2 data nodes started</li>
<li>Set permissions for data path of 1 of the nodes (node-2) to read only for all</li>
<li>Create a new index (eg. say with only primary shards)</li>
<li>All shards allocated for node-2 for this new index are unassigned (as expected since it cannot write to its file system), and primary shards for the good node are allocated successfully:</li>
</ol>
<p dir="auto">testindex               3     p      STARTED       0   130b 127.0.0.1 node1<br>
testindex               4     p      UNASSIGNED<br>
testindex               2     p      UNASSIGNED<br>
testindex               1     p      STARTED       0   130b 127.0.0.1 node1<br>
testindex               0     p      UNASSIGNED</p>
<p dir="auto">However, the master node is not recovering from this scenario well.  It keeps trying to load the shards onto node 2 ... pretty much perpetually as long as node2 is started with a read only file system.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2016-08-03 11:45:04,927][WARN ][gateway                  ] [node1] [testindex][2]: failed to list shard for shard_started on node [Cr5kuAANQAi7dShxyW83Jg]
FailedNodeException[Failed node [Cr5kuAANQAi7dShxyW83Jg]]; nested: RemoteTransportException[[node2][127.0.0.1:9301][internal:gateway/local/started_shards[n]]]; nested: ElasticsearchException[failed to load started shards]; nested: NotSerializableExceptionWrapper[access_denied_exception: /Users/User/ELK/ElasticStack_2_0/elasticsearch-2.3.2_node2/data/my-application1/nodes/0/indices/testindex/2/_state];
    at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:206)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$800(TransportNodesAction.java:106)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$2.handleException(TransportNodesAction.java:179)
    at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:212)
    at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:202)
    at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:136)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: RemoteTransportException[[node2][127.0.0.1:9301][internal:gateway/local/started_shards[n]]]; nested: ElasticsearchException[failed to load started shards]; nested: NotSerializableExceptionWrapper[access_denied_exception: /Users/User/ELK/ElasticStack_2_0/elasticsearch-2.3.2_node2/data/my-application1/nodes/0/indices/testindex/2/_state];
Caused by: ElasticsearchException[failed to load started shards]; nested: NotSerializableExceptionWrapper[access_denied_exception: /Users/User/ELK/ElasticStack_2_0/elasticsearch-2.3.2_node2/data/my-application1/nodes/0/indices/testindex/2/_state];
    at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:154)
    at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:59)
    at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:92)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:230)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:226)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)"><pre class="notranslate"><code class="notranslate">[2016-08-03 11:45:04,927][WARN ][gateway                  ] [node1] [testindex][2]: failed to list shard for shard_started on node [Cr5kuAANQAi7dShxyW83Jg]
FailedNodeException[Failed node [Cr5kuAANQAi7dShxyW83Jg]]; nested: RemoteTransportException[[node2][127.0.0.1:9301][internal:gateway/local/started_shards[n]]]; nested: ElasticsearchException[failed to load started shards]; nested: NotSerializableExceptionWrapper[access_denied_exception: /Users/User/ELK/ElasticStack_2_0/elasticsearch-2.3.2_node2/data/my-application1/nodes/0/indices/testindex/2/_state];
    at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:206)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$800(TransportNodesAction.java:106)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$2.handleException(TransportNodesAction.java:179)
    at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:212)
    at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:202)
    at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:136)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: RemoteTransportException[[node2][127.0.0.1:9301][internal:gateway/local/started_shards[n]]]; nested: ElasticsearchException[failed to load started shards]; nested: NotSerializableExceptionWrapper[access_denied_exception: /Users/User/ELK/ElasticStack_2_0/elasticsearch-2.3.2_node2/data/my-application1/nodes/0/indices/testindex/2/_state];
Caused by: ElasticsearchException[failed to load started shards]; nested: NotSerializableExceptionWrapper[access_denied_exception: /Users/User/ELK/ElasticStack_2_0/elasticsearch-2.3.2_node2/data/my-application1/nodes/0/indices/testindex/2/_state];
    at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:154)
    at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:59)
    at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:92)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:230)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:226)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
</code></pre></div>
<p dir="auto">And these tasks keep getting added and re-added to the pending tasks to no end:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;tasks&quot;: [
    {
      &quot;insert_order&quot;: 141469,
      &quot;priority&quot;: &quot;HIGH&quot;,
      &quot;source&quot;: &quot;cluster_reroute(async_shard_fetch)&quot;,
      &quot;executing&quot;: true,
      &quot;time_in_queue_millis&quot;: 1,
      &quot;time_in_queue&quot;: &quot;1ms&quot;
    },
    {
      &quot;insert_order&quot;: 141470,
      &quot;priority&quot;: &quot;HIGH&quot;,
      &quot;source&quot;: &quot;cluster_reroute(async_shard_fetch)&quot;,
      &quot;executing&quot;: false,
      &quot;time_in_queue_millis&quot;: 0,
      &quot;time_in_queue&quot;: &quot;0s&quot;
    }
  ]
}"><pre class="notranslate"><code class="notranslate">{
  "tasks": [
    {
      "insert_order": 141469,
      "priority": "HIGH",
      "source": "cluster_reroute(async_shard_fetch)",
      "executing": true,
      "time_in_queue_millis": 1,
      "time_in_queue": "1ms"
    },
    {
      "insert_order": 141470,
      "priority": "HIGH",
      "source": "cluster_reroute(async_shard_fetch)",
      "executing": false,
      "time_in_queue_millis": 0,
      "time_in_queue": "0s"
    }
  ]
}
</code></pre></div>
<p dir="auto">It keeps going forever until node 2 is stopped and the underlying file system is addressed.</p>
<p dir="auto">Once node2 is started back up with a writable data path, then you end up with a red index for it does not go and retry the allocation there.</p>
<p dir="auto">testindex               3     p      STARTED       0   130b 127.0.0.1 node1<br>
testindex               4     p      UNASSIGNED<br>
testindex               2     p      UNASSIGNED<br>
testindex               1     p      STARTED       0   130b 127.0.0.1 node1<br>
testindex               0     p      UNASSIGNED</p>
<p dir="auto">Seems like there is an opportunity to handle this better:</p>
<ul dir="auto">
<li>Should the master be retrying forever and spewing large amounts of exceptions and keep performing async_shard_fetch actions?  Or should it give up at some point?</li>
<li>Note that even though currently the master is retrying async_shard_fetch forever, if I fix the permission issues against the data folder on node-2 while this is happening, the master will stop complaining about the read permission issues as if things are good now.  However, the shards remain unassigned so it doesn't actually go and try to start them up on node-2.  Even after a full cluster restart, the testindex remains red with the unassigned shards.</li>
</ul>
<p dir="auto">The combo analysis plugin is not maintained anymore and it's quite usefull and necessery for multilingual indexes. This plugin should be moved to core and maintened by elastic.</p>
<p dir="auto"><a href="https://github.com/yakaz/elasticsearch-analysis-combo">https://github.com/yakaz/elasticsearch-analysis-combo</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="116802517" data-permission-text="Title is private" data-url="https://github.com/yakaz/elasticsearch-analysis-combo/issues/21" data-hovercard-type="pull_request" data-hovercard-url="/yakaz/elasticsearch-analysis-combo/pull/21/hovercard" href="https://github.com/yakaz/elasticsearch-analysis-combo/pull/21">yakaz/elasticsearch-analysis-combo#21</a></p>
<p dir="auto">Thank you</p>