Web crawling is just like most things: 80% of the results for 20% of the work. It&#x27;s always the last mile that takes the most significant cost and engineering effort.<p>as your index and scale grow you bump into the really difficult problems:<p>1. How do you handle so many DNS requests&#x2F;sec without overloading upstream servers?<p>2. How do you discover and determine the quality of new links? It&#x27;s only a matter of time until your crawler hits a treasure trove of spam domains.<p>3. How do you store, update, and access an index that&#x27;s exponentially growing?<p>Just some ideas.