Crawling a quarter billion webpages in 40 hours (2012)
This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping.<p>You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this:<p>- SPAs with front end only rendering are hard to scrape with a traditional http library.<p>- Anti bot/scraping technology fingerprints browsers, looks at request patterns, and browser behaviour to try and detect and block bots.<p>Using a "real browser" is often advised as a way around these issues.<p>However from my experience you should avoid headless browser crawling until it is absolutely necessary, I have found:<p>- Headless browser scraping is between 10x and 100x more resource intensive, even if you carefully block requests and cache resources.<p>- Most SPAs now have some level of server side rendering, and often that includes having a handy JSON in the returned document that contains the data you actually want.<p>- Advanced browser fingerprinting is vanishingly rare. At most I have seen detection of user agent strings and comparing them to http headers and the order of them. If you make your http lib look like a current browser you are %99.9 of the way there.