title: How is HDF5 different from a folder with files?
body: <p>I'm working on an <a href="https://github.com/abstractfactory/openmetadata" rel="noreferrer">open source project</a> dealing with adding metadata to folders. The provided (Python) API lets you browse and access metadata like it was just another folder. Because it is just another folder.</p>
  
  <pre><code>\folder\.meta\folder\somedata.json
  </code></pre>
  
  <p>Then I came across <a href="http://www.hdfgroup.org/HDF5/" rel="noreferrer">HDF5</a> and its derivation <a href="http://www.alembic.io/" rel="noreferrer">Alembic</a>.</p>
  
  <p>Reading up on HDF5 in the book <a href="http://shop.oreilly.com/product/0636920030249.do" rel="noreferrer">Python and HDF5</a> I was looking for benefits to using it compared to using files in folders, but most of what I came across spoke about the benefits of a hierarchical file-format in terms of its simplicity in adding data via its API:</p>
  
  <pre><code>&gt;&gt;&gt; import h5py
  &gt;&gt;&gt; f = h5py.File("weather.hdf5")
  &gt;&gt;&gt; f["/15/temperature"] = 21
  </code></pre>
  
  <p>Or its ability to read only certain parts of it upon request (e.g. random access), and parallel execution of a single HDF5 file (e.g. for multiprocessing)</p>
  
  <p>You could mount HDF5 files, <a href="https://github.com/zjttoefs/hdfuse5" rel="noreferrer">https://github.com/zjttoefs/hdfuse5</a></p>
  
  <p>It even boasts a strong yet simple foundation concept of <strong>Groups</strong> and <strong>Datasets</strong> which from wiki reads:</p>
  
  <ul>
  <li>Datasets, which are multidimensional arrays of a homogeneous type </li>
  <li>Groups, which are container structures which can hold datasets and
  other groups</li>
  </ul>
  
  <p>Replace <strong>Dataset</strong> with <strong>File</strong> and <strong>Group</strong> with <strong>Folder</strong> and the whole feature-set sounds to me like what files in folders are already fully capable of doing.</p>
  
  <p>For every benefit I came across, not one stood out as being exclusive to HDF5.</p>
  
  <p>So my question is, if I were to give you one HDF5 file and one folder with files, both with identical content, in which scenario would HDF5 be better suited?</p>
  
  <p>Edit:</p>
  
  <p>Having gotten some responses about the portability of HDF5.</p>
  
  <p>It sounds lovely and all, but I still haven't been given an example, a scenario, in which an HDF5 would out-do a folder with files. Why would someone consider using HDF5 when a folder is readable on any computer, any file-system, over a network, supports "parallel I/O", is readable by humans without an HDF5 interpreter.</p>
  
  <p>I would go as far as to say, a folder with files is far more portable than any HDF5.</p>
  
  <p>Edit 2:</p>
  
  <p>Thucydides411 just gave an example of a scenario where portability matters.
  <a href="https://stackoverflow.com/a/28512028/478949">https://stackoverflow.com/a/28512028/478949</a></p>
  
  <p>I think what I'm taking away from the answers in this thread is that HDF5 is well suited for when you need the organisational structure of files and folders, like in the example scenario above, with lots (millions) small (~1 byte) data structures; like individual numbers or strings. That it makes up for what file-systems lack by providing a "sub file-system" favouring the small and many as opposed to few and large.</p>
  
  <p>In computer graphics, we use it to store geometric models and arbitrary data about individual vertices which seems to align quite well with it's use in the scientific community.</p>