<p dir="auto">I decided to reindex my data to take advantage of <code class="notranslate">doc_values</code>, but one of 30 indices (~120m docs in each) got less documents after reindexing. I reindexed again and docs disappeared again.</p>
<p dir="auto">Then I bisected the problem to specific docs and found that some docs in source index has duplicate ids.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -s &quot;http://web245:9200/statistics-20141110/_search?pretty&amp;q=_id:1jC2LxTjTMS1KHCn0Prf1w&quot;"><pre class="notranslate"><code class="notranslate">curl -s "http://web245:9200/statistics-20141110/_search?pretty&amp;q=_id:1jC2LxTjTMS1KHCn0Prf1w"
</code></pre></div>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;took&quot; : 1156,
  &quot;timed_out&quot; : false,
  &quot;_shards&quot; : {
    &quot;total&quot; : 5,
    &quot;successful&quot; : 5,
    &quot;failed&quot; : 0
  },
  &quot;hits&quot; : {
    &quot;total&quot; : 2,
    &quot;max_score&quot; : 1.0,
    &quot;hits&quot; : [ {
      &quot;_index&quot; : &quot;statistics-20141110&quot;,
      &quot;_type&quot; : &quot;events&quot;,
      &quot;_id&quot; : &quot;1jC2LxTjTMS1KHCn0Prf1w&quot;,
      &quot;_score&quot; : 1.0,
      &quot;_source&quot;:{&quot;@timestamp&quot;:&quot;2014-11-10T14:30:00+0300&quot;,&quot;@key&quot;:&quot;client_belarussia_msg_sended_from_mutual__22_1&quot;,&quot;@value&quot;:&quot;149&quot;}
    }, {
      &quot;_index&quot; : &quot;statistics-20141110&quot;,
      &quot;_type&quot; : &quot;events&quot;,
      &quot;_id&quot; : &quot;1jC2LxTjTMS1KHCn0Prf1w&quot;,
      &quot;_score&quot; : 1.0,
      &quot;_source&quot;:{&quot;@timestamp&quot;:&quot;2014-11-10T14:30:00+0300&quot;,&quot;@key&quot;:&quot;client_belarussia_msg_sended_from_mutual__22_1&quot;,&quot;@value&quot;:&quot;149&quot;}
    } ]
  }
}"><pre class="notranslate">{
  <span class="pl-ent">"took"</span> : <span class="pl-c1">1156</span>,
  <span class="pl-ent">"timed_out"</span> : <span class="pl-c1">false</span>,
  <span class="pl-ent">"_shards"</span> : {
    <span class="pl-ent">"total"</span> : <span class="pl-c1">5</span>,
    <span class="pl-ent">"successful"</span> : <span class="pl-c1">5</span>,
    <span class="pl-ent">"failed"</span> : <span class="pl-c1">0</span>
  },
  <span class="pl-ent">"hits"</span> : {
    <span class="pl-ent">"total"</span> : <span class="pl-c1">2</span>,
    <span class="pl-ent">"max_score"</span> : <span class="pl-c1">1.0</span>,
    <span class="pl-ent">"hits"</span> : [ {
      <span class="pl-ent">"_index"</span> : <span class="pl-s"><span class="pl-pds">"</span>statistics-20141110<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"_type"</span> : <span class="pl-s"><span class="pl-pds">"</span>events<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"_id"</span> : <span class="pl-s"><span class="pl-pds">"</span>1jC2LxTjTMS1KHCn0Prf1w<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"_score"</span> : <span class="pl-c1">1.0</span>,
      <span class="pl-ent">"_source"</span>:{<span class="pl-ent">"@timestamp"</span>:<span class="pl-s"><span class="pl-pds">"</span>2014-11-10T14:30:00+0300<span class="pl-pds">"</span></span>,<span class="pl-ent">"@key"</span>:<span class="pl-s"><span class="pl-pds">"</span>client_belarussia_msg_sended_from_mutual__22_1<span class="pl-pds">"</span></span>,<span class="pl-ent">"@value"</span>:<span class="pl-s"><span class="pl-pds">"</span>149<span class="pl-pds">"</span></span>}
    }, {
      <span class="pl-ent">"_index"</span> : <span class="pl-s"><span class="pl-pds">"</span>statistics-20141110<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"_type"</span> : <span class="pl-s"><span class="pl-pds">"</span>events<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"_id"</span> : <span class="pl-s"><span class="pl-pds">"</span>1jC2LxTjTMS1KHCn0Prf1w<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"_score"</span> : <span class="pl-c1">1.0</span>,
      <span class="pl-ent">"_source"</span>:{<span class="pl-ent">"@timestamp"</span>:<span class="pl-s"><span class="pl-pds">"</span>2014-11-10T14:30:00+0300<span class="pl-pds">"</span></span>,<span class="pl-ent">"@key"</span>:<span class="pl-s"><span class="pl-pds">"</span>client_belarussia_msg_sended_from_mutual__22_1<span class="pl-pds">"</span></span>,<span class="pl-ent">"@value"</span>:<span class="pl-s"><span class="pl-pds">"</span>149<span class="pl-pds">"</span></span>}
    } ]
  }
}</pre></div>
<p dir="auto">Here are two indices, source and destination:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="health status index                  pri rep docs.count docs.deleted store.size pri.store.size
green  open   statistics-20141110      5   0  116217042            0     12.3gb         12.3gb
green  open   statistics-20141110-dv   5   1  116216507            0     32.3gb         16.1gb"><pre class="notranslate"><code class="notranslate">health status index                  pri rep docs.count docs.deleted store.size pri.store.size
green  open   statistics-20141110      5   0  116217042            0     12.3gb         12.3gb
green  open   statistics-20141110-dv   5   1  116216507            0     32.3gb         16.1gb
</code></pre></div>
<p dir="auto">Segments of problematic index:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="index               shard prirep ip            segment generation docs.count docs.deleted    size size.memory committed searchable version compound
statistics-20141110 0     p      192.168.0.190 _gga         21322   14939669            0   1.6gb     4943008 true      true       4.9.0   false
statistics-20141110 0     p      192.168.0.190 _isc         24348   10913518            0   1.1gb     4101712 true      true       4.9.0   false
statistics-20141110 1     p      192.168.0.245 _7i7          9727    7023269            0   766mb     2264472 true      true       4.9.0   false
statistics-20141110 1     p      192.168.0.245 _i01         23329   14689581            0   1.5gb     4788872 true      true       4.9.0   false
statistics-20141110 2     p      192.168.1.212 _9wx         12849    8995444            0 987.7mb     3326288 true      true       4.9.0   false
statistics-20141110 2     p      192.168.1.212 _il1         24085   13205585            0   1.4gb     4343736 true      true       4.9.0   false
statistics-20141110 3     p      192.168.1.212 _8pc         11280   10046395            0     1gb     4003824 true      true       4.9.0   false
statistics-20141110 3     p      192.168.1.212 _hwt         23213   13226096            0   1.3gb     4287544 true      true       4.9.0   false
statistics-20141110 4     p      192.168.2.88  _91i         11718    8328558            0 909.2mb     2822712 true      true       4.9.0   false
statistics-20141110 4     p      192.168.2.88  _hms         22852   14848927            0   1.5gb     4777472 true      true       4.9.0   false"><pre class="notranslate"><code class="notranslate">index               shard prirep ip            segment generation docs.count docs.deleted    size size.memory committed searchable version compound
statistics-20141110 0     p      192.168.0.190 _gga         21322   14939669            0   1.6gb     4943008 true      true       4.9.0   false
statistics-20141110 0     p      192.168.0.190 _isc         24348   10913518            0   1.1gb     4101712 true      true       4.9.0   false
statistics-20141110 1     p      192.168.0.245 _7i7          9727    7023269            0   766mb     2264472 true      true       4.9.0   false
statistics-20141110 1     p      192.168.0.245 _i01         23329   14689581            0   1.5gb     4788872 true      true       4.9.0   false
statistics-20141110 2     p      192.168.1.212 _9wx         12849    8995444            0 987.7mb     3326288 true      true       4.9.0   false
statistics-20141110 2     p      192.168.1.212 _il1         24085   13205585            0   1.4gb     4343736 true      true       4.9.0   false
statistics-20141110 3     p      192.168.1.212 _8pc         11280   10046395            0     1gb     4003824 true      true       4.9.0   false
statistics-20141110 3     p      192.168.1.212 _hwt         23213   13226096            0   1.3gb     4287544 true      true       4.9.0   false
statistics-20141110 4     p      192.168.2.88  _91i         11718    8328558            0 909.2mb     2822712 true      true       4.9.0   false
statistics-20141110 4     p      192.168.2.88  _hms         22852   14848927            0   1.5gb     4777472 true      true       4.9.0   false
</code></pre></div>
<p dir="auto">The only thing that happened with index besides indexing is optimizing to 2 segments per shard.</p>
<p dir="auto"><strong>Elasticsearch version</strong>: 1.7.5</p>
<p dir="auto"><strong>JVM version</strong>:</p>
<p dir="auto"><strong>OS version</strong>: CentOS 7.2.1511</p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>:  I have been working on this for a few days now, and found similar issues, but nothing that matched what I am seeing.  Every time I restart elasticsearch the shards begin to initialize until this point via _cluster/health</p>
<p dir="auto">{<br>
"cluster_name": "elasticsearch",<br>
"status": "red",<br>
"timed_out": false,<br>
"number_of_nodes": 1,<br>
"number_of_data_nodes": 1,<br>
"active_primary_shards": 5458,<br>
"active_shards": 5458,<br>
"relocating_shards": 0,<br>
"initializing_shards": 0,<br>
"unassigned_shards": 5574,<br>
"delayed_unassigned_shards": 0,<br>
"number_of_pending_tasks": 0,<br>
"number_of_in_flight_fetch": 0<br>
}</p>
<p dir="auto">I then attempt to manually reroute shards, which fails.  I have run reroute?explain and receive the following:</p>
<p dir="auto">"explanations": [<br>
{<br>
"command": "allocate",<br>
"parameters": {<br>
"index": "logstash-2016.03.11-mr",<br>
"shard": 0,<br>
"node": "loggy",<br>
"allow_primary": true<br>
},<br>
"decisions": [<br>
{<br>
"decider": "same_shard",<br>
"decision": "NO",<br>
"explanation": "shard cannot be allocated on same node [3OedVZN9QVKU9Hw7tn2h2w] it already exists on"<br>
},<br>
{<br>
"decider": "filter",<br>
"decision": "YES",<br>
"explanation": "node passes include/exclude/require filters"<br>
},<br>
{<br>
"decider": "replica_after_primary_active",<br>
"decision": "YES",<br>
"explanation": "primary is already active"<br>
},<br>
{<br>
"decider": "throttling",<br>
"decision": "YES",<br>
"explanation": "below shard recovery limit of [4]"<br>
},<br>
{<br>
"decider": "enable",<br>
"decision": "YES",<br>
"explanation": "allocation disabling is ignored"<br>
},<br>
{<br>
"decider": "disable",<br>
"decision": "YES",<br>
"explanation": "allocation disabling is ignored"<br>
},<br>
{<br>
"decider": "awareness",<br>
"decision": "YES",<br>
"explanation": "no allocation awareness enabled"<br>
},<br>
{<br>
"decider": "shards_limit",<br>
"decision": "YES",<br>
"explanation": "total shard limit disabled: [-1] &lt;= 0"<br>
},<br>
{<br>
"decider": "node_version",<br>
"decision": "YES",<br>
"explanation": "target node version [1.7.5] is same or newer than source node version [1.7.5]"<br>
},<br>
{<br>
"decider": "disk_threshold",<br>
"decision": "YES",<br>
"explanation": "only a single node is present"<br>
},<br>
{<br>
"decider": "snapshot_in_progress",<br>
"decision": "YES",<br>
"explanation": "shard not primary or relocation disabled"<br>
}<br>
]<br>
}<br>
]</p>
<p dir="auto">I have a single node cluster and also attempted to adjust the number of replicas to 0 per another post I found, which did not resolve the issue.  Physically inspecting the file system also doesn't appear to have duplicates.  This started about a week ago, and as far as I know nothing major changed on the system.  I do think we hit a limit as I also had to adjust the fs filemax, but this issue is still not correct after multiple attempts to restart and re-index.</p>
<p dir="auto">I have also enabled routing allocation.  My current _cluster/settings are:</p>
<p dir="auto">{<br>
"acknowledged": true,<br>
"persistent": {},<br>
"transient": {<br>
"cluster": {<br>
"routing": {<br>
"allocation": {<br>
"enable": "true"<br>
}<br>
}<br>
}<br>
}<br>
}</p>
<p dir="auto">Running _cat/nodes?v only returns a single node as I would expect.</p>
<p dir="auto">_cat/allocation?v returns 20% free space:<br>
shards disk.used disk.avail disk.total disk.percent host                               ip            node<br>
5458   315.2gb     77.3gb    392.6gb           80 vcosdevlog01.[SNIPPED] [X.X.X.X] loggy<br>
5574                                                                                               UNASSIGNED</p>
<p dir="auto">Any help would be greatly appreciated!</p>