How to Crawl the Web with Scrapy
We have to crawl about 60-80k news websites per day [0].I've spent about 1 month to test how scrapy could be a fit for our purposes. And, quite surprisingly, it was hard to design a distributed web crawler. Scrapy is great for those in-the-middle tasks where you need to crawl a bit + process data on the go.We ended up just using requests to crawl the web. Then post-process the web pages in the next step.Many thanks to Zyte [1] (ex-ScrapingHub) for open-sourcing so many wonderful tools for us. I've spoke to Zyte's CEO, and was really fascinated how he still being a dev person while running such a big company.[0] https://newscatcherapi.com/news-api
[1] https://www.zyte.com/