I'm crawling a bunch of web pages using Python's request library, but occasionally the crawler will stumble upon an absolutely mammoth page, be it a PDF or video or otherwise gargantuan file.  Is there a good way to limit the maximum size of file it will download?
The urlopen object has a method info() which gives all kinds of useful header information, including Content-Length
Occassionally this is not correctly set but should be in most cases and will help
Cap download size with Python requests library