<p dir="auto">Hi,<br>
Elasticsearch 5.0.0 now 5.1.1<br>
Running JVM 1.8.102 and then updated to JVM 1.8.112 - problem occurs on both.<br>
Windows 2008 R2</p>
<p dir="auto">I updated from 2.4.1 to 5.0 then to 5.1.1 since I've moved to 5.0 I've been having nodes drop out of the cluster. I have a 6 data nodes, each one exhibits the exact same problem but at random times. My ingest and master nodes do not exhibit this issue, but as soon as I enable a ingest node as a data node it starts to drop. I never had this issue before when on 2.4.1.</p>
<p dir="auto">Logging doesn't show anything other than the node left and joined, I turned the logging up to debug but the same problem logging stops outputting anything, even the usual 10s marvel metrics shipping stops logging. I'm not able to bring up the hot threads via 9200 as that connection times out. Even trying to do a jstack dump hangs until the java process starts to normalize. The 100% cpu spike can last a few seconds to a few minutes, but it does recover on it's own. Disk I/O is at nothing. I've reduced my cluster down to 2 data nodes made them mangers and removed 95% of my indexes to accommodate only the current daily indexes, the problem persists. No searches are occurring. I'm now starting to disable all other windows services to see if there's some conflict.</p>
<p dir="auto">I only have process explorer, and the thread dump from as soon as the jstack was able to connect and dump.</p>
<p dir="auto">These are all the threads in the java.exe when running at 100%<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/10899305/21236561/879a68ce-c2c9-11e6-8af0-fee168727fdd.png"><img src="https://cloud.githubusercontent.com/assets/10899305/21236561/879a68ce-c2c9-11e6-8af0-fee168727fdd.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">Some stack views from the TIDs if these will help any.<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/10899305/21236815/7eb1a7f8-c2ca-11e6-89e1-5431434c10ac.png"><img src="https://cloud.githubusercontent.com/assets/10899305/21236815/7eb1a7f8-c2ca-11e6-89e1-5431434c10ac.png" alt="10224" style="max-width: 100%;"></a><br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/10899305/21236824/877da74c-c2ca-11e6-89c4-6e3d62e3b793.png"><img src="https://cloud.githubusercontent.com/assets/10899305/21236824/877da74c-c2ca-11e6-89c4-6e3d62e3b793.png" alt="2936" style="max-width: 100%;"></a></p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/10899305/21236574/9f9f9156-c2c9-11e6-98f9-b51943c8bc6a.png"><img src="https://cloud.githubusercontent.com/assets/10899305/21236574/9f9f9156-c2c9-11e6-98f9-b51943c8bc6a.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><a href="https://github.com/elastic/elasticsearch/files/655492/threadDumps.txt">threadDumps.txt</a></p>
<p dir="auto">Assume a 2-node cluster with an index with 1 primary and 1 replica. The index has <code class="notranslate">auto_expand_replicas</code> set <code class="notranslate">0-all</code>. The second node drops, which leads to the first node automatically resetting the <code class="notranslate">number_of_replicas</code> to 0. This is a process that's triggered in a delayed fashion. <code class="notranslate">MetaDataUpdateSettingsService</code> listens on cluster state changed events and submits a cluster state update task to adjust <code class="notranslate">number_of_replicas</code> when it detects that the number of data nodes is smaller/greater than the number of currently-configured replicas. Assume this update successfully completes. When the second node rejoins, it sees a shard routing table that has all shards active (= primary only) and starts deleting it's local shard copy. Shortly thereafter (maybe a few milliseconds?) the first node updates the cluster state by auto-expanding the number of replicas back to one. The second node however has deleted the data and needs to resync the complete shard.</p>