<p dir="auto">The new explain API for detailing heap usage can be a huge help in debugging large Elasticsearch systems - for performance or crashes</p>
<p dir="auto">Obviously if you can add the ES's structures to this functionality , all the better</p>
<p dir="auto"><a href="https://issues.apache.org/jira/browse/LUCENE-5949" rel="nofollow">https://issues.apache.org/jira/browse/LUCENE-5949</a></p>
<p dir="auto">A primary shard was stuck in RELOCATION state.  Recovery API shows that all files have completed at 100%.  The stage shows that it is TRANSLOG.  But it has been sitting there for &gt; 15 hours.</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="  {
        &quot;id&quot;: 0,
        &quot;type&quot;: &quot;RELOCATION&quot;,
        &quot;stage&quot;: &quot;TRANSLOG&quot;,
        &quot;primary&quot;: true,
        &quot;start_time&quot;: &quot;16443.3d&quot;,
        &quot;start_time_in_millis&quot;: 1420704589802,
        &quot;stop_time&quot;: &quot;0s&quot;,
        &quot;stop_time_in_millis&quot;: 0,
        &quot;total_time&quot;: &quot;15.1h&quot;,
        &quot;total_time_in_millis&quot;: 54706836,
        &quot;source&quot;: {
          &quot;id&quot;: &quot;0eltfic0RdmC9v-Yb17Wzw&quot;,
          &quot;host&quot;: &quot;elasticsearch1.localdomain&quot;,
          &quot;transport_address&quot;: &quot;inet[/IP:9300]&quot;,
          &quot;ip&quot;: &quot;127.0.1.1&quot;,
          &quot;name&quot;: &quot;elasticsearch1.localdomain&quot;
        },
        &quot;target&quot;: {
          &quot;id&quot;: &quot;ctCRm_huQsSBoTobhmqJdg&quot;,
          &quot;host&quot;: &quot;elasticsearch8.localdomain&quot;,
          &quot;transport_address&quot;: &quot;inet[/IP:9300]&quot;,
          &quot;ip&quot;: &quot;127.0.1.1&quot;,
          &quot;name&quot;: &quot;elasticsearch8.localdomain&quot;
        },
        &quot;index&quot;: {
          &quot;files&quot;: {
            &quot;total&quot;: 318,
            &quot;reused&quot;: 0,
            &quot;recovered&quot;: 318,
            &quot;percent&quot;: &quot;100.0%&quot;,
            &quot;details&quot;: [
             ... list of files
            ]
          },
          &quot;bytes&quot;: {
            &quot;total&quot;: 58111147927,
            &quot;reused&quot;: 0,
            &quot;recovered&quot;: 58111147927,
            &quot;percent&quot;: &quot;100.0%&quot;
          },
          &quot;total_time&quot;: &quot;0s&quot;,
          &quot;total_time_in_millis&quot;: 0
        },
        &quot;translog&quot;: {
          &quot;recovered&quot;: 24685532,
          &quot;total_time&quot;: &quot;0s&quot;,
          &quot;total_time_in_millis&quot;: 0
        },
        &quot;start&quot;: {
          &quot;check_index_time&quot;: &quot;0s&quot;,
          &quot;check_index_time_in_millis&quot;: 0,
          &quot;total_time&quot;: &quot;0s&quot;,
          &quot;total_time_in_millis&quot;: 0
        }
      }
"><pre class="notranslate">  {
        <span class="pl-ent">"id"</span>: <span class="pl-c1">0</span>,
        <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>RELOCATION<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"stage"</span>: <span class="pl-s"><span class="pl-pds">"</span>TRANSLOG<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"primary"</span>: <span class="pl-c1">true</span>,
        <span class="pl-ent">"start_time"</span>: <span class="pl-s"><span class="pl-pds">"</span>16443.3d<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"start_time_in_millis"</span>: <span class="pl-c1">1420704589802</span>,
        <span class="pl-ent">"stop_time"</span>: <span class="pl-s"><span class="pl-pds">"</span>0s<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"stop_time_in_millis"</span>: <span class="pl-c1">0</span>,
        <span class="pl-ent">"total_time"</span>: <span class="pl-s"><span class="pl-pds">"</span>15.1h<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"total_time_in_millis"</span>: <span class="pl-c1">54706836</span>,
        <span class="pl-ent">"source"</span>: {
          <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>0eltfic0RdmC9v-Yb17Wzw<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"host"</span>: <span class="pl-s"><span class="pl-pds">"</span>elasticsearch1.localdomain<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"transport_address"</span>: <span class="pl-s"><span class="pl-pds">"</span>inet[/IP:9300]<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"ip"</span>: <span class="pl-s"><span class="pl-pds">"</span>127.0.1.1<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>elasticsearch1.localdomain<span class="pl-pds">"</span></span>
        },
        <span class="pl-ent">"target"</span>: {
          <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>ctCRm_huQsSBoTobhmqJdg<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"host"</span>: <span class="pl-s"><span class="pl-pds">"</span>elasticsearch8.localdomain<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"transport_address"</span>: <span class="pl-s"><span class="pl-pds">"</span>inet[/IP:9300]<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"ip"</span>: <span class="pl-s"><span class="pl-pds">"</span>127.0.1.1<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>elasticsearch8.localdomain<span class="pl-pds">"</span></span>
        },
        <span class="pl-ent">"index"</span>: {
          <span class="pl-ent">"files"</span>: {
            <span class="pl-ent">"total"</span>: <span class="pl-c1">318</span>,
            <span class="pl-ent">"reused"</span>: <span class="pl-c1">0</span>,
            <span class="pl-ent">"recovered"</span>: <span class="pl-c1">318</span>,
            <span class="pl-ent">"percent"</span>: <span class="pl-s"><span class="pl-pds">"</span>100.0%<span class="pl-pds">"</span></span>,
            <span class="pl-ent">"details"</span>: [
             <span class="pl-ii">... list of files</span>
            ]
          },
          <span class="pl-ent">"bytes"</span>: {
            <span class="pl-ent">"total"</span>: <span class="pl-c1">58111147927</span>,
            <span class="pl-ent">"reused"</span>: <span class="pl-c1">0</span>,
            <span class="pl-ent">"recovered"</span>: <span class="pl-c1">58111147927</span>,
            <span class="pl-ent">"percent"</span>: <span class="pl-s"><span class="pl-pds">"</span>100.0%<span class="pl-pds">"</span></span>
          },
          <span class="pl-ent">"total_time"</span>: <span class="pl-s"><span class="pl-pds">"</span>0s<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"total_time_in_millis"</span>: <span class="pl-c1">0</span>
        },
        <span class="pl-ent">"translog"</span>: {
          <span class="pl-ent">"recovered"</span>: <span class="pl-c1">24685532</span>,
          <span class="pl-ent">"total_time"</span>: <span class="pl-s"><span class="pl-pds">"</span>0s<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"total_time_in_millis"</span>: <span class="pl-c1">0</span>
        },
        <span class="pl-ent">"start"</span>: {
          <span class="pl-ent">"check_index_time"</span>: <span class="pl-s"><span class="pl-pds">"</span>0s<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"check_index_time_in_millis"</span>: <span class="pl-c1">0</span>,
          <span class="pl-ent">"total_time"</span>: <span class="pl-s"><span class="pl-pds">"</span>0s<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"total_time_in_millis"</span>: <span class="pl-c1">0</span>
        }
      }
</pre></div>
<p dir="auto">Running a reroute command with no post body got it to unstuck:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XPOST 'localhost:9200/_cluster/reroute?pretty&amp;explain=true'"><pre class="notranslate"><code class="notranslate">curl -XPOST 'localhost:9200/_cluster/reroute?pretty&amp;explain=true'
</code></pre></div>
<p dir="auto">Not sure how it got into this state (no log entries related to recovery of this shard in the data or master logs) and why we have to run an empty reroute request to get it unstuck.</p>