title: HBase cluster with corrupt region file on HDFS
body: <p>We have this HBase cluster: 30+ nodes, 48 tables, 40+TB on HDFS level, replication factor 2. Due to disk failure on two nodes, we have a corrupt file on HDFS.</p>
  
  <h2>Current HDFS status</h2>
  
  <p>Excerpt of <code>hdfs fsck /</code> output, which shows a corrupt HBase region file:</p>
  
  <pre><code>/user/hbase/table_foo_bar/295cff9c67379c1204a6ddd15808af0b/n/ae0fdf7d0fa24ad1914ca934d3493e56: 
   CORRUPT blockpool BP-323062689-192.168.12.45-1357244568924 block blk_9209554458788732793
  /user/hbase/table_foo_bar/295cff9c67379c1204a6ddd15808af0b/n/ae0fdf7d0fa24ad1914ca934d3493e56:
   MISSING 1 blocks of total size 134217728 B
  
    CORRUPT FILES:        1
    MISSING BLOCKS:       1
    MISSING SIZE:         134217728 B
    CORRUPT BLOCKS:       1
  
  The filesystem under path '/' is CORRUPT
  </code></pre>
  
  <p>The lost data is not recoverable (the disks are broken).</p>
  
  <h2>Current HBase status</h2>
  
  <p>According to HBase on the other hand, everything is fine and dandy</p>
  
  <p><code>hbase hbck</code> says:</p>
  
  <pre><code>Version: 0.94.6-cdh4.4.0
  ...
   table_foo_bar is okay.
     Number of regions: 1425
     Deployed on:  ....
  ...
  0 inconsistencies detected.
  Status: OK   
  </code></pre>
  
  <p>Moreover, it seems that we can still query data from the non-lost blocks of the corrupt region file (as far as I think I was able to check based on the start and end row key of the region).</p>
  
  <h2>Next steps</h2>
  
  <ul>
  <li>Because the file block data is not recoverable,  it seems the only option is to remove the complete corrupt file (with <code>hadoop fs -rm</code> or  <code>hadoop fsck -delete /</code>). This will "fix" corruption at the HDFS level.</li>
  <li>However, I'm afraid removing the HDFS file will introduce corruption at the HBase level as a complete region file will be gone</li>
  <li>I considered <code>hadoop fsck -move /</code> to move the corrupt file to <code>/lost+found</code> and see how HBase would take that, but moving to <code>/lost+found</code> is <a href="http://www.swiss-scalability.com/2013/08/hadoop-fsck-move-does-not-only-move.html" rel="noreferrer">not as reversible as it seems</a>, so I'm hesitant about that as well</li>
  </ul>
  
  <p>Concrete questions:</p>
  
  <p>Should I just remove the file? (Losing the data corresponding to that region is reasonably fine for us.) What bad things happen when you manually remove a HBase region file in HDFS? Does it just remove the data or would it introduce ugly metadata corruption in HBase that also have to be taken care of?</p>
  
  <p>Or can we actually leave the situation as-is, which seems to work at the moment (HBase is not complaining about/seeing corruption)?</p>