How to Crawl the Web with Scrapy
We have to crawl about 60-80k news websites per day [0].<p>I've spent about 1 month to test how scrapy could be a fit for our purposes. And, quite surprisingly, it was hard to design a distributed web crawler. Scrapy is great for those in-the-middle tasks where you need to crawl a bit + process data on the go.<p>We ended up just using requests to crawl the web. Then post-process the web pages in the next step.<p>Many thanks to Zyte [1] (ex-ScrapingHub) for open-sourcing so many wonderful tools for us. I've spoke to Zyte's CEO, and was really fascinated how he still being a dev person while running such a big company.<p>[0] <a href="https://newscatcherapi.com/news-api" rel="nofollow">https://newscatcherapi.com/news-api</a>
[1] <a href="https://www.zyte.com/" rel="nofollow">https://www.zyte.com/</a>