<p dir="auto"><strong>Problem Description</strong><br>
I faced an issue when upgrading my elasticsearch 2.3.3 to elasticsearch 5.1.1 by upgrading from apt repository. When elasticsearch 5.1.1 was upgraded and I tried to start it, Elasticsearch did not start but in the STDOUT it never gave a failure. Instead, in STDOUT it gave an OK message wwhich signifies that elasticsearch has started succesfully.</p>
<p dir="auto"><strong>Problem Issue Identification</strong><br>
After debugging it, I found the issue causing it by traversing through the /var/log/elasticsearch/elasticsearch.log file. The issue was caused by using elasticsearch-head plugin which was installed in ES-2.3.3 but as it is not supported in ES-5.1.1, elasticsearch failed to start.</p>
<p dir="auto"><strong>Actual Behaviour</strong><br>
As an end-user I think we should not get an OK message if elasticsearch fails to start and it should be mentioned in the Installation of Elasticsearch documentation (<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html" rel="nofollow">https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html</a>).</p>
<p dir="auto"><strong>Note: This issue will occur for every plugins which were earlier supported prior to ES-5.0.x version.</strong></p>
<p dir="auto">Steps To Reproduce:</p>
<ol dir="auto">
<li>Install any ES version prior to ES-5.x.x</li>
<li>Install Elasticsearch-head in previous ES version (Validate its installation as it will be listed under $ES_HOME/plugins directory)</li>
<li>Upgrade ES to ES-5.1.1</li>
<li>Add ES as a service</li>
<li>Run ES as a service. You will get following STDOUT console on command line:-</li>
</ol>
<p dir="auto">root@ubuntu:/home/yuvraj# sudo service elasticsearch start<br>
Starting Elasticsearch Server                                                                         [2016-12-11T12:07:50,609][WARN ][o.e.c.l.LogConfigurator  ] ignoring unsupported logging configuration file [/etc/elasticsearch/logging.yml], logging is configured via [/etc/elasticsearch/log4j2.properties]<br>
[ OK ]</p>
<p dir="auto">Please find required details as mentioned below -</p>
<p dir="auto"><strong>Elasticsearch version</strong>: Elasticsearch-5.1.1</p>
<p dir="auto"><strong>Plugins installed</strong>: elasticsearch-head</p>
<p dir="auto"><strong>JVM version</strong>: Oracle java version "1.8.0_91"</p>
<p dir="auto"><strong>OS version</strong>: Ubuntu 14.04</p>
<p dir="auto"><strong>Provide logs (if relevant)</strong>: <a href="https://github.com/elastic/elasticsearch/files/644320/elasticsearch-error-captured.txt">elasticsearch-error-captured.txt</a></p>
<p dir="auto">In the current implementation of zen discovery the cluster state broadcasting by the master does not scale too well as every change in the cluster state results in the cluster state being sent to every single node by the master.</p>
<p dir="auto">This can lead to quite some traffic, in an cluster with 500 nodes and 15000 shards, even when the cluster state by itself has only 1MB, the master node will sent a total of 500MB per cluster state update.</p>
<p dir="auto">So it should be possible to only send a diff of the changes to the nodes instead of the full cluster state. In case the nodes recognize that they missed an update, they could query the master node to resend the full cluster state to them.</p>
<p dir="auto">Another possibility would be a tiered distribution of the state, where the master node does not directly send the updates to every single node, but through other nodes which forward the state to the other nodes in the cluster. A gossipping protocol could here be used.</p>
<p dir="auto">If the clusterstate does not need to be distributed in realtime, one could also throttle the propagation of the state, so that when changes come in rapid succession, the changes can be merged before the state has been sent out to every node.</p>
<p dir="auto">Any ideas?</p>