title: efficient numpy.fromfile on zipped files?
body: <p>I have some large (even gzipped around 10GB) files, which contain an ASCII header and then in principle numpy.recarrays of about 3MB each, we call them "events". My first approach looked like this:</p>
  
  <pre><code>f = gzip.GzipFile(filename)
  f.read(10000) # fixed length ascii header
  event_dtype = np.dtype([
          ('Id', '&gt;u4'),                # simplified
          ('UnixTimeUTC', '&gt;u4', 2), 
          ('Data', '&gt;i2', (1600,1024) ) 
          ])
  event = np.fromfile( f, dtype = event_dtype, count=1 )
  </code></pre>
  
  <p>However, this is not possible, since np.fromfile needs a real FILE object, because it really makes low level calls (found a pretty old ticket <a href="https://github.com/numpy/numpy/issues/1103" rel="nofollow noreferrer">https://github.com/numpy/numpy/issues/1103</a>).</p>
  
  <p>So as I understand I have to do it like this:</p>
  
  <pre><code>s = f.read( event_dtype.itemsize )
  event = np.fromstring(s, dtype=event_dtype, count=1)
  </code></pre>
  
  <p>And yes, it works! But isn't this awfully inefficient? Isn't the mem for s allocated, and garbage collected for every event? 
  On my laptop I reach something like 16 events/s, i.e. ~50MB/s </p>
  
  <p>I wonder if anybody knows a smart way, to allocate the mem once and then let numpy read directly into that mem.</p>
  
  <p>Btw. I'm a physicist, so ... well still a newbie in this business.</p>