document: Recrawl scheduling based on information longevity It is crucial for a web crawler to distinguish between ephemeral and persistent content . Ephemeral content ( e.g. , quote of the day ) is usually not worth crawling , because by the time it reaches the index it is no longer representative of the web page from which it was acquired . On the other hand , content that persists across multiple page updates ( e.g. , recent blog postings ) may be worth acquiring , because it matches the page 's true content for a sustained period of time . In this paper we characterize the longevity of information found on the web , via both empirical measurements and a generative model that coincides with these measurements . We then develop new recrawl scheduling policies that take longevity into account . As we show via experiments over real web data , our policies obtain better freshness at lower cost , compared with previous approaches .
doc_bio_tags: O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O