Rotating IPs and user agents using Scrapy CrawlerRunner
If anyone else is wondering, there isn't a way to do this when using the CrawlerRunner function. I just had to set up a Scrapy project which didn't end up being as hard as I thought it would be. Theres plenty of info and "hello world" tutorials to follow online.
Good luck!
I haven't set up my spider as a Scrapy project, so don't have the settings.py file for settings. But I still want to implement as many methods as I can to avoid being blocked/blacklisted. Is there any way to add these rotations and settings outside of a Scrapy project and only within the CrawlerRunner function???
Thanks!