title: Store large text corpus in Python
body: <p>I am trying to build a large text corpus from the <a href="http://dumps.wikimedia.org/" rel="nofollow">Wikipedia dump</a>. 
  I represent every article as a <em>document</em> object consisting of:</p>
  
  <ul>
  <li>the original text: a string</li>
  <li>the preprocessed text: a list of tuples, where each tuple contains a (stemmed) word and the position of the word in the original text</li>
  <li>some additional information like title and author</li>
  </ul>
  
  <p>I am searching for an efficient way to save these objects to disk. The following operations should be possible:</p>
  
  <ul>
  <li>adding a new document</li>
  <li>accessing documents via an ID</li>
  <li>iterate over all documents</li>
  </ul>
  
  <p>It is not necessary to remove an object once it was added.</p>
  
  <p>I could imagine the following methods:</p>
  
  <ul>
  <li><strong>Serializing each article to a separate file, for example using pickle:</strong> the downside here are probably lots of operating system calls</li>
  <li><strong>Store all documents to a single xml file or blocks of documents to several xml files:</strong> Storing the list that represents the preprocessed document in xml format uses a lot of overhead and I think its quite slow to read a list from xml</li>
  <li><strong>Using an existing package for storing the corpus:</strong> I found <a href="https://pypi.python.org/pypi/Corpora/1.0" rel="nofollow">Corpora</a> package, which seems to be very fast and efficient, but it only supports storing strings plus a header including metadata. Simply putting the preprocessed text into the header makes it run incredibly slow.</li>
  </ul>
  
  <p>What would be a good way to do this? Maybe a package for that purpose, which i have not found until now?</p>