title: How to deserialize 1GB of objects into Python faster than cPickle?
body: <p>We've got a Python-based web server that unpickles a number of large data files on startup using <code>cPickle</code>. The data files (pickled using <code>HIGHEST_PROTOCOL</code>) are around 0.4 GB on disk and load into memory as about 1.2 GB of Python objects -- this takes about <strong>20 seconds</strong>. We're using Python 2.6 on 64-bit Windows machines.</p>
  
  <p>The bottleneck is certainly not disk (it takes less than 0.5s to actually read that much data), but memory allocation and object creation (there are millions of objects being created). We want to reduce the 20s to decrease startup time.</p>
  
  <p>Is there any way to deserialize more than 1GB of objects into Python much faster than <code>cPickle</code> (like 5-10x)? Because the execution time is bound by memory allocation and object creation, I presume using another unpickling technique such as JSON wouldn't help here.</p>
  
  <p>I know some interpreted languages have a way to save their entire memory image as a disk file, so they can load it back into memory all in one go, without allocation/creation for each object. Is there a way to do this, or achieve something similar, in Python?</p>