Crawling a quarter billion webpages in 40 hours (2012)
This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping.You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this:- SPAs with front end only rendering are hard to scrape with a traditional http library.- Anti bot/scraping technology fingerprints browsers, looks at request patterns, and browser behaviour to try and detect and block bots.Using a "real browser" is often advised as a way around these issues.However from my experience you should avoid headless browser crawling until it is absolutely necessary, I have found:- Headless browser scraping is between 10x and 100x more resource intensive, even if you carefully block requests and cache resources.- Most SPAs now have some level of server side rendering, and often that includes having a handy JSON in the returned document that contains the data you actually want.- Advanced browser fingerprinting is vanishingly rare. At most I have seen detection of user agent strings and comparing them to http headers and the order of them. If you make your http lib look like a current browser you are %99.9 of the way there.