How to crawl a quarter billion webpages in 40 hours (2012)
For those curious like I was how much it would cost to scrape the entire internet with the method and numbers provided:250.000.000 pages come in at $580There are 1.8b websites according to http://www.internetlivestats.com/total-number-of-websites/Lets say on average each site has 10 pages (you have a dozen of huge blogs vs tens of thousands of onepagers), that would put the number at 18 billion pages.Following that logic would mean the total web is 72 times larger than what was scraped in this test.So for a mere $41.760 you too can bootstrap your own Google! ;-)