I'm scraping some 26,000 items in a web site from its API via Scrapy. I use a rotating proxy. When I run the spider it starts working well for a while. But then the rate of 400 responses increases up to a point that almost 95% of are refused.
When I switch from my current Wi-Fi and connect to another Wi-Fi, for example my phone's hotspot, the rate of 200 responses peak again to be ultimately slowed down again. I set the retry times to 50 which extremely prolongs the scraping process which takes up to 16 hours. I think there must be a way to reduce these negative responses.
So, the question is, despite the fact that I use a proxy, why would changing the connected Wi-Fi station increases my rates of 200 responses?
Thank you
Depending on the proxy software you are using, it may be setting the HTTP headers X-Forwarded-For and X-Real-IP, which can be used to see your IP even if you are using a proxy. Chaning your wifi, however, changes really device's IP, and therefore the IP in X-Forwarded-For and X-Real-IP. I don't know what proxy you are using, but check if there are any options or modules to spoof/unset the X-Forwarded-For and X-Real-IP headers.
400 responses switch to 200 if I change the wifi, despite the use of proxy