Ask HN: What are best tools for web scraping?
If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler.<p>Scrapy also has the ability to pause and restart crawls [1], run the crawlers distributed [2] etc. It is my goto option.<p>[0] <a href="https://scrapy.org/" rel="nofollow">https://scrapy.org/</a><p>[1] <a href="https://doc.scrapy.org/en/latest/topics/jobs.html" rel="nofollow">https://doc.scrapy.org/en/latest/topics/jobs.html</a><p>[2] <a href="https://github.com/rmax/scrapy-redis" rel="nofollow">https://github.com/rmax/scrapy-redis</a>