Checking source code in a scrapy response
Check the view-source with (Ctrl-U in Chrome). Chrome tools will not always line up with the html source. Probably due to the JavaScript on the page.
I have made a pretty big spider that basically extracts the data from an amazon product page.
The problem is that sometimes, no data comes back when I extract. After that happens I check the URL that was processed and, following the xpath with a chrome tool, the data is in fact there.
I know that what me and the Chrome tool sees is not the same as what the spider processes so, is it there any way to actually see the source code the spider is trying to extract from? and will the XPath I make with the chrome tool's help be trustworthy?