title: How can I work with Gzip files which contain extra data?
body: <p>I'm writing a script which will work with data coming from instrumentation as gzip streams.  In about 90% of cases, the <code>gzip</code> module works perfectly, but some of the streams cause it to produce <code>IOError: Not a gzipped file</code>.  If the gzip header is removed and the deflate stream fed directly to <code>zlib</code>, I instead get <code>Error -3 while decompressing data: incorrect header check</code>.  After about half a day of banging my head against the wall, I discovered that the streams which are having problems contain a seemingly-random number of extra bytes (which are not part of the gzip data) appended to the end.</p>
  
  <p>It strikes me as odd that Python cannot work with these files for two reasons:</p>
  
  <ol>
  <li>Both Gzip and 7zip are able to open these "padded" files without issue.  (Gzip produces the message <code>decompression OK, trailing garbage ignored</code>, 7zip succeeds silently.)</li>
  <li><p>Both the Gzip and Python docs seem to indicate that this should work: (emphasis mine)</p>
  
  <p><a href="http://www.gzip.org/format.txt" rel="noreferrer">Gzip's format.txt</a>:</p>
  
  <blockquote>
    <p>It must be possible to
    detect the end of the compressed data with any compression method,
    regardless of the actual size of the compressed data. <strong>In particular,
    the decompressor must be able to detect and skip extra data appended
    to a valid compressed file</strong> on a record-oriented file system, or when
    the compressed data can only be read from a device in multiples of a
    certain block size.</p>
  </blockquote>
  
  <p><a href="http://docs.python.org/library/gzip.html#gzip.GzipFile" rel="noreferrer">Python's gzip.GzipFile`</a>:</p>
  
  <blockquote>
    <p>Calling a <code>GzipFile</code> object’s <code>close()</code> method does not close <em>fileobj</em>, <strong>since you might wish to append more material after the compressed data</strong>. This also allows you to pass a <code>StringIO</code> object opened for writing as <em>fileobj</em>, and retrieve the resulting memory buffer using the <code>StringIO</code> object’s <code>getvalue()</code> method.</p>
  </blockquote>
  
  <p><a href="http://docs.python.org/library/zlib.html#zlib.Decompress.unused_data" rel="noreferrer">Python's <code>zlib.Decompress.unused_data</code></a>:</p>
  
  <blockquote>
    <p>A string which contains any bytes past the end of the compressed data. That is, this remains <code>""</code> until the last byte that contains compression data is available. <strong>If the whole string turned out to contain compressed data, this is <code>""</code>, the empty string.</strong></p>
    
    <p>The only way to determine where a string of compressed data ends is by actually decompressing it. This means that when compressed data is contained part of a larger file, you can only find the end of it by <strong>reading data and feeding it followed by some non-empty string into a decompression object’s <code>decompress()</code></strong> method until the <code>unused_data</code> attribute is no longer the empty string.</p>
  </blockquote></li>
  </ol>
  
  <p>Here are the four approaches I've tried.  (These examples are Python 3.1, but I've tested 2.5 and 2.7 and had the same problem.)</p>
  
  <pre><code># approach 1 - gzip.open
  with gzip.open(filename) as datafile:
      data = datafile.read()
  
  # approach 2 - gzip.GzipFile
  with open(filename, "rb") as gzipfile:
      with gzip.GzipFile(fileobj=gzipfile) as datafile:
          data = datafile.read()
  
  # approach 3 - zlib.decompress
  with open(filename, "rb") as gzipfile:
      data = zlib.decompress(gzipfile.read()[10:])
  
  # approach 4 - zlib.decompressobj
  with open(filename, "rb") as gzipfile:
      decompressor = zlib.decompressobj()
      data = decompressor.decompress(gzipfile.read()[10:])
  </code></pre>
  
  <p>Am I doing something wrong?</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>Okay, while the problem with <code>gzip</code> seems to be a bug in the module, my <code>zlib</code> problems are self-inflicted.  ;-)</p>
  
  <p>While digging into <code>gzip.py</code> I realized what I was doing wrong — by default, <code>zlib.decompress</code> et al. expect zlib-wrapped streams, not bare deflate streams.  By passing in a negative value for <code>wbits</code>, you can tell <code>zlib</code> to skip the zlib header and decrompress the raw stream.  Both of these work:</p>
  
  <pre><code># approach 5 - zlib.decompress with negative wbits
  with open(filename, "rb") as gzipfile:
      data = zlib.decompress(gzipfile.read()[10:], -zlib.MAX_WBITS)
  
  # approach 6 - zlib.decompressobj with negative wbits
  with open(filename, "rb") as gzipfile:
      decompressor = zlib.decompressobj(-zlib.MAX_WBITS)
      data = decompressor.decompress(gzipfile.read()[10:])
  </code></pre>