How to crawl a quarter billion webpages in 40 hours (2012)
For those curious like I was how much it would cost to scrape the entire internet with the method and numbers provided:<p>250.000.000 pages come in at $580<p>There are 1.8b websites according to <a href="http://www.internetlivestats.com/total-number-of-websites/" rel="nofollow">http://www.internetlivestats.com/total-number-of-websites/</a><p>Lets say on average each site has 10 pages (you have a dozen of huge blogs vs tens of thousands of onepagers), that would put the number at 18 billion pages.<p>Following that logic would mean the total web is 72 times larger than what was scraped in this test.<p>So for a mere $41.760 you too can bootstrap your own Google! ;-)