document: User-centric Web crawling Search engines are the primary gateways of information access on the Web today . Behind the scenes , search engines crawl the Web to populate a local indexed repository of Web pages , used to answer user search queries . In an aggregate sense , the Web is very dynamic , causing any repository of Web pages to become out of date over time , which in turn causes query answer quality to degrade . Given the considerable size , dynamicity , and degree of autonomy of the Web as a whole , it is not feasible for a search engine to maintain its repository exactly synchronized with the Web . In this paper we study how to schedule Web pages for selective ( re ) downloading into a search engine repository . The scheduling objective is to maximize the quality of the user experience for those who query the search engine . We begin with a quantitative characterization of the way in which the discrepancy between the content of the repository and the current content of the live Web impacts the quality of the user experience . This characterization leads to a user-centric metric of the quality of a search engine 's local repository . We use this metric to derive a policy for scheduling Web page ( re ) downloading that is driven by search engine usage and free of exterior tuning parameters . We then focus on the important subproblem of scheduling refreshing of Web pages already present in the repository , and show how to compute the priorities efficiently . We provide extensive empirical comparisons of our user-centric method against prior Web page refresh strategies , using real Web data . Our results demonstrate that our method requires far fewer resources to maintain same search engine quality level for users , leaving substantially more resources available for incorporating new Web pages into the search repository .
doc_bio_tags: O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O