I am trying to scrape a website using Scrapy framework in python. But i am getting the captchas. The server implements the bot detection using Distil netwrok bot detection. Is there anyway i can work around with it?
I personally drown it in proxies. 1 proxy for 4 requests before blocked, then I change proxy. I've several tens of thousands of free proxies, so it's not a big problem. But it's not very fast, so I set concurrency to 1k or about that.
Working of the distil networks bot detection