title: Best splittable compression for Hadoop input = bz2?
body: <p>We've realized a bit too late that archiving our files in GZip format for Hadoop processing isn't such a great idea. GZip isn't splittable, and for reference, here are the problems which I won't repeat:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/2078850/very-basic-question-about-hadoop-and-compressed-input-files">Very basic question about Hadoop and compressed input files</a></li>
  <li><a href="https://stackoverflow.com/questions/5630245/hadoop-gzip-compressed-files">Hadoop gzip compressed files</a></li>
  <li><a href="https://stackoverflow.com/questions/7388436/hadoop-gzip-input-file-using-only-one-mapper">Hadoop gzip input file using only one mapper</a></li>
  <li><a href="https://stackoverflow.com/questions/6511255/why-cant-hadoop-split-up-a-large-text-file-and-then-compress-the-splits-using-g">Why can&#39;t hadoop split up a large text file and then compress the splits using gzip?</a></li>
  </ul>
  
  <p>My question is: is BZip2 the best archival compression that will allow a single archive file to be processed in parallel by Hadoop? Gzip is definitely not, and from my reading LZO has some problems.</p>