title: Ways to store small files in Hadoop HDFS other than HAR or Sequence Files + doubts about them
body: <p>I have read <strong>lots of blog entries</strong> and articles about the "Small Files problem in hadoop", but a lot of them <strong>simply seem to be a copy-paste</strong> of the previous. Furthermore they all seem a little bit dated, and the last ones (2015ish) describe anyway what <a href="http://blog.cloudera.com/blog/2009/02/the-small-files-problem/" rel="nofollow">this cloudera blog</a> did in the early 2009.<br>
  <em>Does this mean no archiving solution has been found in 6 years?</em></p>
  
  <p>Here is the reason of my research: <strong>I need to move and catalogue files</strong> as they come, in different numbers, sometimes even singlely, and <strong>then store them in HDFS</strong>.<br>
  These files will be <strong>later</strong> be accessed and <strong>returned in a web service layer</strong> (must be fast), to be opened and seen by people or softwares.<br>
  <strong>The files may be videos, images, documents</strong>, whatever and need to be accessed later using an ID I produce with the Java class <code>UUID</code>.<br>
  <strong>The choice to use hdfs is completey personal</strong> of my PM, as I've proposed HBase to compensate the lack of indexing in HDFS (although I'm not sure it is an optimal solution), but he has asked me to look anyway outisde of HBase in case of having to deal with bigger files (so far the biggest among 1000 has been 2MB, but we expect 1Gb videos).<br>
  As far as I have understood, the small files problem happen when you use MapReduce jobs, for memory consumption, but I was wondering:<br>
  <em>Does it really matter how many files are there in HDFS if I am using Spark to extract them? Or if I am using webhdfs/v1/ ? Or Java?</em>  </p>
  
  <p>Talking about storing a <em>group</em> of small files, <strong>so far I've found three main solutions</strong>, all of which are quite inconvenient in production environment:</p>
  
  <ul>
  <li><strong>HAR</strong>: looks fantastic with the <strong>indexed file extraction</strong>, but the fact that I <strong>cannot append or add new files</strong> is quite troublesome. <em>Does the opening and recreation of HARs weigh a lot on the system?</em></li>
  <li><strong>Sequence Files</strong> have the opposite pros and cons: you <strong>can append files, but they're not indexed</strong>, so there is a O(n) look-up time. <em>Is it worth it?</em> </li>
  <li><strong>Merge them: impossible</strong> to do in my case.  </li>
  </ul>
  
  <p><em>Is there some new technology I'm missing out regarding this common problem? Something on the lines of Avro or Parquet for files?</em></p>