title: Python: slow read & write for millions of small files
body: <blockquote>
    <p><strong>Conclusion:</strong>
    It seems that HDF5 is the way to go for my purposes. Basically "<em>HDF5 is a data model, library, and file format for storing and managing data.</em>" and is designed to handle incredible amounts of data. It has a Python module called python-tables. (The link is in the answer below)</p>
    
    <p><em>HDF5 does the job done 1000% better in saving tons and tons of data. Reading/modifying the data from 200 million rows is a pain though, so that's the next problem to tackle.</em></p>
  </blockquote>
  
  <hr>
  
  <p>I am building directory tree which has tons of subdirectories and files. There are about 10 million files spread around a hundred thousand directories. Each file is under 32 subdirectories.</p>
  
  <p>I have a python script that builds this filesystem and reads &amp; writes those files. The problem is that when I reach more than a million files, the read and write methods become extremely slow.</p>
  
  <p>Here's the function I have that reads the contents of a file (the file contains an integer string), adds a certain number to it, then writes it back to the original file.</p>
  
  <pre><code>def addInFile(path, scoreToAdd):
      num = scoreToAdd
      try:
          shutil.copyfile(path, '/tmp/tmp.txt')
          fp = open('/tmp/tmp.txt', 'r')
          num += int(fp.readlines()[0])
          fp.close()
      except:
          pass
      fp = open('/tmp/tmp.txt', 'w')
      fp.write(str(num))
      fp.close()
      shutil.copyfile('/tmp/tmp.txt', path)
  </code></pre>
  
  <ul>
  <li>Relational databases seem too slow for accessing these data, so I opted for a filesystem approach.</li>
  <li>I previously tried performing linux console commands for these but it was way slower. </li>
  <li>I copy the file to a temporary file first then access/modify it then copy it back because i found this was faster than directly accessing the file.</li>
  <li>Putting all the files into 1 directory (in reiserfs format) caused too much slowdown when accessing the files.</li>
  </ul>
  
  <p>I think the cause of the slowdown is because there're tons of files. Performing this function 1000 times clocked at less than a second.. but now it's reaching 1 minute.</p>
  
  <p>How do you suggest I fix this? Do I change my directory tree structure? </p>
  
  <p><strong>All I need is to quickly access each file in this very huge pool of files*</strong></p>