Scrapy - use multiple IP Addresses for a host
You can just set your DNS names manually in your hosts file. On windows this can be found at C:\Windows\System32\Drivers\etc\hosts and on Linux in /etc/hosts
Wasn't able to find anything in the docs/SO relating to my question.
So basically I'm crawling a website with 8 or so subdomains
They are all using Akamai/CDN.
My question is if I can find Ips of a few different Akamai data centres can, I somehow explicitly say this subdomain should use this ip for the host name etc.. So basically override the auto dns resolving...
As this would allow greater efficiency and I would imagine less likely to be throttled as I'd be distributing the crawling.
Thanks