<p dir="auto">I followed the guide mentioned here <a href="https://www.elastic.co/guide/en/x-pack/current/api-java.html" rel="nofollow">https://www.elastic.co/guide/en/x-pack/current/api-java.html</a> and all in all everything worked within the IDE.</p>
<p dir="auto">Our application is spring boot based, so we end up creating fat jars for deployment using prebuild maven plugins. The problem is, when we run the jar file we get the following error</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.elasticsearch.client.Client]: Factory method 'esClient' threw exception; nested exception is java.lang.NoClassDefFoundError: org/elasticsearch/xpack/XPackPlugin
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
	... 41 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/xpack/XPackPlugin
	at org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.&lt;init&gt;(PreBuiltXPackTransportClient.java:50)
	at org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.&lt;init&gt;(PreBuiltXPackTransportClient.java:46)"><pre class="notranslate"><code class="notranslate">Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.elasticsearch.client.Client]: Factory method 'esClient' threw exception; nested exception is java.lang.NoClassDefFoundError: org/elasticsearch/xpack/XPackPlugin
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
	... 41 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/xpack/XPackPlugin
	at org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.&lt;init&gt;(PreBuiltXPackTransportClient.java:50)
	at org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.&lt;init&gt;(PreBuiltXPackTransportClient.java:46)
</code></pre></div>
<p dir="auto">Here's the dependencies we added:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" &lt;dependency&gt;
            &lt;groupId&gt;org.elasticsearch.client&lt;/groupId&gt;
            &lt;artifactId&gt;x-pack-transport&lt;/artifactId&gt;
            &lt;version&gt;5.0.0&lt;/version&gt;
        &lt;/dependency&gt;"><pre class="notranslate"><code class="notranslate"> &lt;dependency&gt;
            &lt;groupId&gt;org.elasticsearch.client&lt;/groupId&gt;
            &lt;artifactId&gt;x-pack-transport&lt;/artifactId&gt;
            &lt;version&gt;5.0.0&lt;/version&gt;
        &lt;/dependency&gt;
</code></pre></div>
<p dir="auto">and here's the code we use to initialize the client (based on the discussion <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="170315623" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/19903" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/19903/hovercard" href="https://github.com/elastic/elasticsearch/issues/19903">#19903</a>)</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Node node = new PluginNode(settings);
        node.start();
        return node.client();
    }

    static class PluginNode extends Node {
        public PluginNode(Settings settings) {
            super(InternalSettingsPreparer.prepareEnvironment(settings, null), Collections.singletonList(Netty4Plugin.class));
        }
    }"><pre class="notranslate"><code class="notranslate">Node node = new PluginNode(settings);
        node.start();
        return node.client();
    }

    static class PluginNode extends Node {
        public PluginNode(Settings settings) {
            super(InternalSettingsPreparer.prepareEnvironment(settings, null), Collections.singletonList(Netty4Plugin.class));
        }
    }
</code></pre></div>
<p dir="auto">also, once in a while we get this in our CI builds as well</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[35,35] cannot find symbol
[ERROR] symbol:   class Netty4Plugin
[ERROR] location: package org.elasticsearch.transport
[ERROR] -&gt; [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] "><pre class="notranslate"><code class="notranslate">[35,35] cannot find symbol
[ERROR] symbol:   class Netty4Plugin
[ERROR] location: package org.elasticsearch.transport
[ERROR] -&gt; [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
</code></pre></div>
<p dir="auto">which doesn't make sense because x-pack-transport by default includes the <code class="notranslate">transport</code> dependency as well</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[INFO] +- org.elasticsearch.client:x-pack-transport:jar:5.0.0:compile
[INFO] |  |  +- org.elasticsearch.client:rest:jar:5.0.0:compile
[INFO] |  |  \- org.elasticsearch.client:sniffer:jar:5.0.0:compile
[INFO] |  \- org.elasticsearch.client:transport:jar:5.0.0:compile"><pre class="notranslate"><code class="notranslate">[INFO] +- org.elasticsearch.client:x-pack-transport:jar:5.0.0:compile
[INFO] |  |  +- org.elasticsearch.client:rest:jar:5.0.0:compile
[INFO] |  |  \- org.elasticsearch.client:sniffer:jar:5.0.0:compile
[INFO] |  \- org.elasticsearch.client:transport:jar:5.0.0:compile
</code></pre></div>
<p dir="auto">I see others are getting some compile time issues as well (on that thread). Is it possible that at runtime xpack transport client doesn't load the right classes?</p>
<p dir="auto">We should add an <code class="notranslate">explain</code> parameter so people can get detailed feedback about why a shard can or cannot be allocated to a node. Something like this:</p>
<h3 dir="auto">For these commands:</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="curl -XPOST 'localhost:9200/_cluster/reroute?explain&amp;pretty' -d '{
  &quot;commands&quot; : [
    {
      &quot;cancel&quot; : {
        &quot;index&quot; : &quot;decide&quot;, &quot;shard&quot; : 0, &quot;node&quot;: &quot;IvpoKRdtRiGrQ_WKtt4_4w&quot;
      }
    },
    {
      &quot;move&quot; : {
        &quot;index&quot; : &quot;decide&quot;, &quot;shard&quot; : 0,
        &quot;from_node&quot; : &quot;IvpoKRdtRiGrQ_WKtt4_4w&quot;, &quot;to_node&quot; : &quot;IvpoKRdtRiGrQ_WKtt4_4w&quot;
      }
    }
  ]
}'"><pre class="notranslate">curl -XPOST <span class="pl-s"><span class="pl-pds">'</span>localhost:9200/_cluster/reroute?explain&amp;pretty<span class="pl-pds">'</span></span> -d <span class="pl-s"><span class="pl-pds">'</span>{</span>
<span class="pl-s">  "commands" : [</span>
<span class="pl-s">    {</span>
<span class="pl-s">      "cancel" : {</span>
<span class="pl-s">        "index" : "decide", "shard" : 0, "node": "IvpoKRdtRiGrQ_WKtt4_4w"</span>
<span class="pl-s">      }</span>
<span class="pl-s">    },</span>
<span class="pl-s">    {</span>
<span class="pl-s">      "move" : {</span>
<span class="pl-s">        "index" : "decide", "shard" : 0,</span>
<span class="pl-s">        "from_node" : "IvpoKRdtRiGrQ_WKtt4_4w", "to_node" : "IvpoKRdtRiGrQ_WKtt4_4w"</span>
<span class="pl-s">      }</span>
<span class="pl-s">    }</span>
<span class="pl-s">  ]</span>
<span class="pl-s">}<span class="pl-pds">'</span></span></pre></div>
<h3 dir="auto">This result:</h3>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;explanations&quot; : [ {
    &quot;command&quot; : &quot;cancel&quot;,
    &quot;parameters&quot; : {
      &quot;index&quot; : &quot;[decide]&quot;,
      &quot;shard&quot; : 0,
      &quot;node&quot; : &quot;IvpoKRdtRiGrQ_WKtt4_4w&quot;,
      &quot;allow_primary&quot; : false
    },
    &quot;decisions&quot; : [ {
      &quot;decider&quot; : &quot;CancelAllocationCommand&quot;,
      &quot;decision&quot; : &quot;NO&quot;,
      &quot;explanation&quot; : &quot;can't cancel [decide][0] on node [Wysper][IvpoKRdtRiGrQ_WKtt4_4w][Xanadu.local][inet[/172.16.1.8:9300]], shard is primary and started&quot;
    } ]
  }, {
    &quot;command&quot; : &quot;move&quot;,
    &quot;parameters&quot; : {
      &quot;index&quot; : &quot;[decide]&quot;,
      &quot;shard&quot; : 0,
      &quot;from_node&quot; : &quot;IvpoKRdtRiGrQ_WKtt4_4w&quot;,
      &quot;to_node&quot; : &quot;IvpoKRdtRiGrQ_WKtt4_4w&quot;
    },
    &quot;decisions&quot; : [ {
      &quot;decider&quot; : &quot;SameShard&quot;,
      &quot;decision&quot; : &quot;NO&quot;,
      &quot;explanation&quot; : &quot;shard cannot be allocated on same node [IvpoKRdtRiGrQ_WKtt4_4w] it already exists on&quot;
    }, {
      &quot;decider&quot; : &quot;Filter&quot;,
      &quot;decision&quot; : &quot;YES&quot;,
      &quot;explanation&quot; : &quot;node passes include/exclude/require filters&quot;
    }, {
      &quot;decider&quot; : &quot;ReplicaAfterPrimaryActive&quot;,
      &quot;decision&quot; : &quot;YES&quot;,
      &quot;explanation&quot; : &quot;shard is primary&quot;
    }, {
      &quot;decider&quot; : &quot;Throttling&quot;,
      &quot;decision&quot; : &quot;YES&quot;,
      &quot;explanation&quot; : &quot;below shard recovery limit of [2]&quot;
    }, {
      &quot;decider&quot; : &quot;Enable&quot;,
      &quot;decision&quot; : &quot;YES&quot;,
      &quot;explanation&quot; : &quot;allocation disabling is ignored&quot;
    }, {
      &quot;decider&quot; : &quot;Disable&quot;,
      &quot;decision&quot; : &quot;YES&quot;,
      &quot;explanation&quot; : &quot;allocation disabling is ignored&quot;
    }, {
      &quot;decider&quot; : &quot;Awareness&quot;,
      &quot;decision&quot; : &quot;YES&quot;,
      &quot;explanation&quot; : &quot;no allocation awareness enabled&quot;
    }, {
      &quot;decider&quot; : &quot;ShardsLimit&quot;,
      &quot;decision&quot; : &quot;YES&quot;,
      &quot;explanation&quot; : &quot;total shard limit disabled: [-1] &lt;= 0&quot;
    }, {
      &quot;decider&quot; : &quot;NodeVersion&quot;,
      &quot;decision&quot; : &quot;YES&quot;,
      &quot;explanation&quot; : &quot;target node version [2.0.0-SNAPSHOT] is same or newer than source node version [2.0.0-SNAPSHOT]&quot;
    }, {
      &quot;decider&quot; : &quot;DiskThreshold&quot;,
      &quot;decision&quot; : &quot;YES&quot;,
      &quot;explanation&quot; : &quot;disk threshold decider disabled&quot;
    }, {
      &quot;decider&quot; : &quot;SnapshotInProgress&quot;,
      &quot;decision&quot; : &quot;YES&quot;,
      &quot;explanation&quot; : &quot;no snapshots are currently running&quot;
    } ]
  } ]
}"><pre class="notranslate">{
  <span class="pl-ent">"explanations"</span> : [ {
    <span class="pl-ent">"command"</span> : <span class="pl-s"><span class="pl-pds">"</span>cancel<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"parameters"</span> : {
      <span class="pl-ent">"index"</span> : <span class="pl-s"><span class="pl-pds">"</span>[decide]<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"shard"</span> : <span class="pl-c1">0</span>,
      <span class="pl-ent">"node"</span> : <span class="pl-s"><span class="pl-pds">"</span>IvpoKRdtRiGrQ_WKtt4_4w<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"allow_primary"</span> : <span class="pl-c1">false</span>
    },
    <span class="pl-ent">"decisions"</span> : [ {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>CancelAllocationCommand<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>NO<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>can't cancel [decide][0] on node [Wysper][IvpoKRdtRiGrQ_WKtt4_4w][Xanadu.local][inet[/172.16.1.8:9300]], shard is primary and started<span class="pl-pds">"</span></span>
    } ]
  }, {
    <span class="pl-ent">"command"</span> : <span class="pl-s"><span class="pl-pds">"</span>move<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"parameters"</span> : {
      <span class="pl-ent">"index"</span> : <span class="pl-s"><span class="pl-pds">"</span>[decide]<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"shard"</span> : <span class="pl-c1">0</span>,
      <span class="pl-ent">"from_node"</span> : <span class="pl-s"><span class="pl-pds">"</span>IvpoKRdtRiGrQ_WKtt4_4w<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"to_node"</span> : <span class="pl-s"><span class="pl-pds">"</span>IvpoKRdtRiGrQ_WKtt4_4w<span class="pl-pds">"</span></span>
    },
    <span class="pl-ent">"decisions"</span> : [ {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>SameShard<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>NO<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>shard cannot be allocated on same node [IvpoKRdtRiGrQ_WKtt4_4w] it already exists on<span class="pl-pds">"</span></span>
    }, {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>Filter<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>YES<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>node passes include/exclude/require filters<span class="pl-pds">"</span></span>
    }, {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>ReplicaAfterPrimaryActive<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>YES<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>shard is primary<span class="pl-pds">"</span></span>
    }, {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>Throttling<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>YES<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>below shard recovery limit of [2]<span class="pl-pds">"</span></span>
    }, {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>Enable<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>YES<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>allocation disabling is ignored<span class="pl-pds">"</span></span>
    }, {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>Disable<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>YES<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>allocation disabling is ignored<span class="pl-pds">"</span></span>
    }, {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>Awareness<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>YES<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>no allocation awareness enabled<span class="pl-pds">"</span></span>
    }, {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>ShardsLimit<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>YES<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>total shard limit disabled: [-1] &lt;= 0<span class="pl-pds">"</span></span>
    }, {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>NodeVersion<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>YES<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>target node version [2.0.0-SNAPSHOT] is same or newer than source node version [2.0.0-SNAPSHOT]<span class="pl-pds">"</span></span>
    }, {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>DiskThreshold<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>YES<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>disk threshold decider disabled<span class="pl-pds">"</span></span>
    }, {
      <span class="pl-ent">"decider"</span> : <span class="pl-s"><span class="pl-pds">"</span>SnapshotInProgress<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"decision"</span> : <span class="pl-s"><span class="pl-pds">"</span>YES<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"explanation"</span> : <span class="pl-s"><span class="pl-pds">"</span>no snapshots are currently running<span class="pl-pds">"</span></span>
    } ]
  } ]
}</pre></div>