title: compressing data with HDFStore
body: <p>I am newbie to pytables and had a question regarding storing compressed pandas DataFrame. My current code is:</p>
  
  <pre><code>import pandas
  # HDF5 file name
  H5name="C:\\MyDir\\MyHDF.h5"
  
  # create HDF5 file
  store=pandas.io.pytables.HDFStore(H5name)
  
  # write a pandas DataFrame to the HDF5 file created
  myDF.to_hdf(H5name,"myDFname",append=True)
  
  # read the pandas DataFrame back from the HDF5 file created
  myDF1=pandas.io.pytables.read_hdf(H5name,"myDFname")
  
  # close the file
  store.close()
  </code></pre>
  
  <p>When I checked the size of the HDF5 created, the size (212kb) was much larger than the original csv file (58kb) I used to create the pandas DataFrame.</p>
  
  <p>So, I tried out compression by (deleting the HDF5 file) and recreating</p>
  
  <pre><code># create HDF5 file
  store=pandas.io.pytables.HDFStore(H5name,complevel=1)
  </code></pre>
  
  <p>and the size of the file created did not change. I tried all <code>complevels</code> from 1 to 9 and the size still remained the same. </p>
  
  <p>I tried to add </p>
  
  <pre><code># create HDF5 file
  store=pandas.io.pytables.HDFStore(H5name,complevel=1,complib="zlib")
  </code></pre>
  
  <p>but it had no change in compression.</p>
  
  <p>What could be the problem?</p>
  
  <p>Also, ideally I would like to use a compression similar to what R does for its save function  (e.g. in my case the 58kb file was saved to a size of 27kb in RData)? Do I need to do any additional serialization in Python to reduce the size?</p>
  
  <p><strong>EDIT:</strong></p>
  
  <p>I am using Python 3.3.3 and Pandas 0.13.1</p>
  
  <p><strong>EDIT:</strong>
  I tried with a larger file 487MB csv file, whose RData <code>size</code> (via R's save function) is 169MB. For larger files, I do see the compressions. Bzip2 gave the best compression of 202MB (level=9) and was the slowest to read/write. Blosc compression (level=9) gave the largest size of 276MB, but was much faster to write/read. </p>
  
  <p>Not sure what R does differently in its <code>save</code> function, but it's both equally fast and much more compressed than any of these compression algos.</p>