I frequently use lxml module in Python to scrape data from some web sites, and I'm comfortable with the module generally. However, when I try to scrape, at times I encounter lxml.etree.XMLSyntaxError: AttValue: " or ' expected error on etree.fromstring() call, but don't usually. I can't clarify how often I see that error, but I think one out of thousands or even tens of thousands times, I encounter the error. When I run the exactly same script immediately after the error occurred and the script stopped, I don't see the error and the script runs well as expected. Why does it spit out an ocasional error? Is there any way to deal with the issue? I have the similar problem when I instantiate urllib2.urlopen() function, but since I haven't seen the error from urllib2 recently, I can't write the exact error message coming from it right now.
Thanks.
I also had the problem that lxml's iterparse() would occasionally throw an AttValue: ' expected in a very unpredictable pattern. I knew that the XML I'm sending in is valid and rerunning the same script would often make it work (or fail at an entirely different point).
In the end, I managed to create a test case that I could rerun and it would immediately either complete or raise an AttValue error in a seemingly random outcome. Here's what I did wrong:
My input to iterparse() was a file-like object I wrote myself (I'm processing an HTTP response stream from requests, but it has to be ungzipped first). When writing the read() method, I cheated and ignored the size argument. Instead, I would just unzip a chunk of compressed bytes of a fixed size and return whatever byte sequence this decompressed to—often much more than the 32k lxml requests!
I suspect that this caused buffer overflows somewhere inside lxml, which led to the above issues. As soon as I stopped returning more bytes than lxml requested, these random errors would go away.
Why does lxml spit out an error at times (but not usual) in Python?