title: Using cPickle to serialize a large dictionary causes MemoryError
body: <p>I'm writing an inverted index for a search engine on a collection of documents. Right now, I'm storing the index as a dictionary of dictionaries. That is, each keyword maps to a dictionary of docIDs->positions of occurrence.</p>
  
  <p>The data model looks something like:
  {word : { doc_name : [location_list] } }</p>
  
  <p>Building the index in memory works fine, but when I try to serialize to disk, I hit a MemoryError. Here's my code:</p>
  
  <pre><code># Write the index out to disk
  serializedIndex = open(sys.argv[3], 'wb')
  cPickle.dump(index, serializedIndex, cPickle.HIGHEST_PROTOCOL)
  </code></pre>
  
  <p>Right before serialization, my program is using about 50% memory (1.6 Gb). As soon as I make the call to cPickle, my memory usage skyrockets to 80% before crashing.</p>
  
  <p>Why is cPickle using so much memory for serialization? Is there a better way to be approaching this problem?</p>