title: Elasticsearch: what does "shard allocation" mean?
body: <p>We encountered a production incident, that Elasticsearch cluster health check returned <code>red</code> status. The health check report shows <code>marvel-2019.06.20</code> has 2 unassigned_shards, which seems the root cause.</p>
  
  <pre><code>curl -XGET 'localhost:9200/_cluster/health?level=indices&amp;pretty'
  
  {
    "cluster_name" : "sap-jam-jam8",
    "status" : "red",
    "timed_out" : false,
    "number_of_nodes" : 2,
    "number_of_data_nodes" : 2,
    "active_primary_shards" : 122,
    "active_shards" : 239,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 7,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "indices" : {
       ...
       ...
       ".marvel-2019.06.20" : {
            "status" : "red",
            "number_of_shards" : 1,
            "number_of_replicas" : 1,
            "active_primary_shards" : 0,
            "active_shards" : 0,
            "relocating_shards" : 0,
            "initializing_shards" : 0,
            "unassigned_shards" : 2
      }
    }
  </code></pre>
  
  <p>we checked the config of Elasticseach, found <code>cluster.routing.allocation</code> has been disabled.</p>
  
  <pre><code>curl -XGET 'localhost:9200/_cluster/settings?pretty'
  {
    "persistent" : { },
    "transient" : {
      "cluster" : {
        "routing" : {
          "allocation" : {
            "enable" : "none"
          }
        }
      }
    }
  }
  </code></pre>
  
  <p>As <a href="https://stackoverflow.com/a/23816954/5117552">this stackoverflow post</a> suggested, we forced a shard to be assigned, and this issue has gone.</p>
  
  <pre><code>curl -XPOST -d '{ "commands" : [ {
    "allocate" : {
         "index" : ".marvel-2014.05.21", 
         "shard" : 0, 
         "node" : "SOME_NODE_HERE",
         "allow_primary":true 
       } 
    } ] }' http://localhost:9200/_cluster/reroute?pretty
  </code></pre>
  
  <hr>
  
  <p>After resolved this incident, I think it's necessary to figure out the basic concept <code>shard allocation</code>. I did some research, but the following questions are still confusing for me.</p>
  
  <h2>1. Why elasticsearch needs to <code>assign shard</code> to other nodes?</h2>
  
  <p>In my case, we have two elasticsearch nodes, A and B. Two shards have already been created in A, and consumed disk space. </p>
  
  <p>When B is not available, why not just active those two shards in server A? </p>
  
  <p>At least it return a <code>yellow</code> health status.</p>
  
  <h2>2. What's the procedures of <code>assign a shard</code>?</h2>
  
  <p>In the first question, we suppose both primary shard and replica has been created in server A. when saying <code>assign shard to B</code>, what does that mean?</p>
  
  <p>Doest that mean copy shard from server A to server B?</p>
  
  <h2>3. How to explain this zero active shard?</h2>
  
  <p>Both primary shard and replicate has been created, but are not active. How is it possible? Besides disk storage, is there other overhead to activate a shard? e.g. Memory?</p>
  
  <pre><code>".marvel-2019.06.20" : {
    "status" : "red",
    "number_of_shards" : 1,
    "number_of_replicas" : 1,
    "active_primary_shards" : 0,
    "active_shards" : 0, // both shards are inactive.
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 2
  }
  </code></pre>
  
  <h2>4. Is the following assumption true?</h2>
  
  <p>To make a shard active, Elasticsearch need do the following steps:</p>
  
  <ul>
  <li>Create a shard.</li>
  <li>Find a server, which has enough disk space and RAM to run it.</li>
  <li>copy this shard from source server to destination server.</li>
  <li>Activate this shard.</li>
  </ul>
  
  <h2>Reference</h2>
  
  <ol>
  <li><p><a href="https://www.elastic.co/blog/red-elasticsearch-cluster-panic-no-longer?nsukey=rugJ0ULHhuAevjmDVw9V7DcHpPoz%2FE8cPo8PzjUhWPGEBBeuPbl7KmJWrKATv16Ao7uTcN4zQ5WefcbmoKSkn%2F2BZyMo9fa7eSrIZDJKfyn7BlIQWFuc5WiS3z2YZ3JRkdWGLrjUAsP9qMORklgpI9ItHwU%2BteEWb%2FjX5oCxf7zMmEnEoZO13%2FiCk1oEltp5eCgo14qYVv7tsGArWgM4Sg%3D%3D" rel="nofollow noreferrer">Elasticsearch blog: Red Elasticsearch Cluster? Panic no longer</a></p></li>
  <li><p><a href="https://stackoverflow.com/a/23816954/5117552">Stack overflow: elasticsearch - what to do with unassigned shards</a></p></li>
  </ol>