It seems like typical crawlers that just download a small number of pages or do very little processing to decide what pages to download are IO limited.
I am curious as to what order of magnitude estimates of sizes relevant data structures, number of stored pages, indexing requirements etc that might actually make CPU the bottleneck?
For example an application might want to calculate some probabilities based on the links found on a page in order to decide what page to crawl next. This function takes O(noOfLinks) and is evaluated N times (at each step)...where N is the number of pages I want to download in one round of crawling.I have to sort and keep track of these probabilities and i have to keep track of a list of O(N) that will eventually be dumped into disk and the index of a search engine. Is it not possible (assuming one machine) that N grows large enough and that storing the pages and manipulating the links gets expensive enough to compete with the IO response?
Not really. It takes I/O to download these additional links, and you're right back to I/O-limited again.
In what scenarios might a web crawler be CPU limited as opposed to IO limited?