Title: What techniques would you use to build an internet scale web crawler&#x2F;search engine.<p>* How would you schedule and manage the crawlers?<p>* How would you collect and parse incoming data?<p>* What would you use to store the data?<p>* How would you process and draw meaning from the data?<p>* What information would be important to build a pagerank system for searching?<p>* What other major components are missing?
Upvote: