Stack Exchange
Error -3 while decompressing data: incorrect header check - urllib2
I am checking for url status with this code: 
h = httplib2.Http()
hdr = {'User-Agent': 'Mozilla/5.0'}
resp = h.request("http://" + url, headers=hdr)
if int(resp[0]['status']) < 400:
    return 'ok'
else:
    return 'bad'

and getting
Error -3 while decompressing data: incorrect header check

the url i am checking is: 
http://www.sueddeutsche.de/wirtschaft/deutschlands-innovationsangst-wir-neobiedermeier-1.2117528
the Exception Location is: 
Exception Location: C:\Python27\lib\site-packages\httplib2-0.9-py2.7.egg\httplib2\__init__.py in _decompressContent, line 403

try:
    encoding = response.get('content-encoding', None)
    if encoding in ['gzip', 'deflate']:
        if encoding == 'gzip':
            content = gzip.GzipFile(fileobj=StringIO.StringIO(new_content)).read()
        if encoding == 'deflate':
            content = zlib.decompress(content) ##<---- error line
        response['content-length'] = str(len(content))
        # Record the historical presence of the encoding in a way the won't interfere.
        response['-content-encoding'] = response['content-encoding']
        del response['content-encoding']
except IOError:
    content = ""

http status is 200 which is ok for my case, but i am getting this error
I actually need only http status, why is it reading the whole content? 

First, if you only want the HTTP status, why are you doing a GET instead of HEAD? This probably isn't related to your problem, but it's still an odd thing to do.

You may have any number of reasons why you choose httplib2, but it's far too easy to get the status code of an HTTP request using the python module requests. Install with the following command:
$ pip install requests

See an extremely simple example below.
In [1]: import requests as rq

In [2]: url = "http://www.sueddeutsche.de/wirtschaft/deutschlands-innovationsangst-wir-neobiedermeier-1.2117528"

In [3]: r = rq.get(url)

In [4]: r
Out[4]: <Response [200]>

Link
Unless you have a considerable constraint that needs httplib2 explicitly, this solves your problem.

He didn't choose urllib2, he's using the third-party library httplib2.

You can also just use any of the modules in the stdlib; none of these seem to have any problem with this site.

It is. @abarnert's explanation about the content being read completely still holds. If you don't want this to be the case, use his suggestion instead.

one of my pains in urlchecking has also to do these cases: httplib2.ServerNotFoundError,UnicodeError, httplib2.RelativeURIError, httplib2.RedirectLimit, httplib2.RedirectMissingLocation, httplib2.ServerNotFoundError, httplib.ResponseNotReady. will requests care about this too?

@doniyor: That depends on what you mean by "reading". It's requesting all of the data, which are flowing over the network to your computer. But if you never access r.data or r.content or anything similar (or the equivalent with other libraries), it may just get discarded from your network buffers when r goes out of scope. Does that count as "reading" or not?

@doniyor: requests doesn't have as many different error types as httplib2. Whether that's better or worse kind of depends on what you're trying to do.

@abarnert i just want to know if the url i am checking is available on the web. but in some cases there too many redirects and stuff which can fool the checking process

@doniyor: Almost all of these libraries make it trivial to do a HEAD request instead of a GET also. For example, with requests, it's just requests.head(…) instead of requests.get(…). The only exception I can think of is urllib2/urllib.request, which requires you to turn the one-liner into two lines to do a different request type.

This may be a bug (or just uncommon design decision) in httplib2. I don't get this problem with urllib2 or httplib in the 2.x stdlib, or urllib.request or http.client in the 3.x stdlib, or the third-party libraries requests, urllib3, or pycurl.
So, is there a reason you need to use this particular library?
If so:

I actually need only http status, why is it reading the whole content?

Well, most HTTP libraries are going to read and parse the whole content, or at least the headers, before returning control. That way they can respond to simple requests about the headers or chunked encoding or MIME envelope or whatever without any delay.
Also, many of them automate things like 100 continue, 302 redirect, various kinds of auth, etc., and there's no way they could do that if they didn't read ahead. In particular, according to the description for httplib2, handling these things automatically is one of the main reasons you should use it in the first place.
Also, the first TCP read is nearly always going to include the headers anyway, so why not read them?
This means that if the headers are invalid, you may get an exception immediately. They may still provide a way to get the status code (or the raw headers, or other information) anyway.

As a side note, if you only want the HTTP status, you should probably send a HEAD request rather than a GET. Unless you're writing and testing a server, you can almost always rely on the fact that, as the RFC says, the status and headers should be identical to what you'd get with GET. In fact, that would almost certainly solve things in this case—if there is no body to decompress, the fact that httplib2 has gotten confused into thinking the body is gzipped when it isn't won't matter anyway.

thank you so much for giving so much effort to explain it to me. appreciate really much

so checking only the headers doesnot go that deep as checking the whole content right? like .head() is lighter than .get() ?

@doniyor: Exactly. HEAD returns the status line and headers that you would have gotten with GET, but without the body you also would have gotten.

ok cool thanks. but thanks again for the whole wonderful information above
