<p dir="auto">If disk encounters disk error ,such as input/output err,  this corrupted disk should be removed automatically from es node if index couldn't be written in for a long time.</p>
<p dir="auto">Currently ops must update config and restart es node.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2016-08-31 10:43:18,568][WARN ][gateway                  ] [10.148.67.11] [slb_itvresult-2016.08.28.05]: failed to write index state
java.nio.file.FileSystemException: /data/slot9/elasticsearch/es02/nodes/0/indices/slb_itvresult-2016.08.28.05: Input/output error
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:308)
        at java.nio.file.Files.createDirectories(Files.java:746)
        at org.elasticsearch.gateway.MetaDataStateFormat.write(MetaDataStateFormat.java:136)
        at org.elasticsearch.gateway.MetaStateService.writeIndex(MetaStateService.java:135)
        at org.elasticsearch.gateway.GatewayMetaState.clusterChanged(GatewayMetaState.java:160)
        at org.elasticsearch.gateway.Gateway.clusterChanged(Gateway.java:185)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)"><pre class="notranslate"><code class="notranslate">[2016-08-31 10:43:18,568][WARN ][gateway                  ] [10.148.67.11] [slb_itvresult-2016.08.28.05]: failed to write index state
java.nio.file.FileSystemException: /data/slot9/elasticsearch/es02/nodes/0/indices/slb_itvresult-2016.08.28.05: Input/output error
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:308)
        at java.nio.file.Files.createDirectories(Files.java:746)
        at org.elasticsearch.gateway.MetaDataStateFormat.write(MetaDataStateFormat.java:136)
        at org.elasticsearch.gateway.MetaStateService.writeIndex(MetaStateService.java:135)
        at org.elasticsearch.gateway.GatewayMetaState.clusterChanged(GatewayMetaState.java:160)
        at org.elasticsearch.gateway.Gateway.clusterChanged(Gateway.java:185)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
</code></pre></div>
<p dir="auto">Hi,</p>
<p dir="auto">This is about documentation probably in tips and tricks section. Sometimes there could be duplicates in documents. Documentation about</p>
<ol dir="auto">
<li>How do we detect duplicates (pre and post index)</li>
<li>How to avoid/remove them?</li>
<li>Is it possible to have a document in more than one type? {"title": "Doc title", "_type": ["type1", "type2"]}</li>
</ol>
<p dir="auto">I used document content (fields) to generate a Hash and stored it with document. Used the computed hash to detect dupplicates and avoided adding them again.</p>