Where does Scrapy actually do the html request?
I believe you can check out scrapy/core/engine.py method _download.
Though I'd suggest you make use of scrapy shell. It will let you execute particular request, inspect response, open response in browser to see what was received by Scrapy. Also with a bit more of tuning you can import your spider in your shell and call a particular method of your spider and put a breakpoint there.
If your spider fails to find any pages then the problem is likely to be with your spider, not the framework.
I am using the Scrapy (Scrapy==1.6.0) library with Python3. I am wondering, where in the code does Scrapy actually do the HTML request? I want to set a breakpoint there so I can see exactly what headers / cookies / urls / and user agent is actually being passed.
Also, where exactly is the response received as well? Right now my spider is failing to find any pages, so I imagine I'm getting either a blank HTML document or a 403 error, however I have no idea where to look to confirm this.
Can anyone familiar with the scrapy library point me to exactly where in code I can check these parameters?