How to crawl a quarter billion webpages in 40 hours (2012)
This was written in 2012, Its even easier these days by using SQS and Cloud Formation. 250 Million is a small number you are better of first going through Common Crawl and then use data from crawls to build a better seed list.<p>Common Crawl now contains repeated crawls conducted  every few months and also urls donated by blekko.<p><a href="https:&#x2F;&#x2F;groups.google.com&#x2F;forum&#x2F;m&#x2F;#!msg&#x2F;common-crawl&#x2F;zexccXgwg4w&#x2F;oV8qeJnawJUJ" rel="nofollow">https:&#x2F;&#x2F;groups.google.com&#x2F;forum&#x2F;m&#x2F;#!msg&#x2F;common-crawl&#x2F;zexccXg...</a>