Synchronicity :-) Just this week I scraped and analyzed only the homepages from the Alexa top 1 million [1] and majestic million [2]. I used rqworker[3] and and a fleet of mini-servers from scaleway[4] to do the scraping. Some results:<p>- 1.8MM successful scrapes
 - 25GB total size (stored in postgres)
 - 1 server to host redis and postgres
 - 9 physical servers for workers (4x core ARM servers) = 36 total cores
 - Peak rate of ~ 100 reqs&#x2F;second achieved across all workers (36 physical cores total)
 - I saw that I could oversubscribe workers to cores by a factor of 2x (72 total workers) to achieve ~75% utilization on each server<p>All in all it was a fun project that provided quite a bit of learning. There are a lot of levers to pull here to make things faster, robust, and more portable. Next steps are to optimize the database and wrap a simple django app around it for exploring data.<p>Or maybe push it futher and try my hand at these 26MM domains?<p>[1] - <a href="http:&#x2F;&#x2F;s3.amazonaws.com&#x2F;alexa-static&#x2F;top-1m.csv.zip" rel="nofollow">http:&#x2F;&#x2F;s3.amazonaws.com&#x2F;alexa-static&#x2F;top-1m.csv.zip</a><p>[2] - <a href="https:&#x2F;&#x2F;blog.majestic.com&#x2F;development&#x2F;majestic-million-csv-daily&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.majestic.com&#x2F;development&#x2F;majestic-million-csv-d...</a><p>[3] - <a href="http:&#x2F;&#x2F;python-rq.org&#x2F;" rel="nofollow">http:&#x2F;&#x2F;python-rq.org&#x2F;</a><p>[4] - <a href="http:&#x2F;&#x2F;scaleway.com" rel="nofollow">http:&#x2F;&#x2F;scaleway.com</a>