document: A large-scale study of the evolution of web pages How fast does the web change ? Does most of the content remain unchanged once it has been authored , or are the documents continuously updated ? Do pages change a little or a lot ? Is the extent of change correlated to any other property of the page ? All of these questions are of interest to those who mine the web , including all the popular search engines , but few studies have been performed to date to answer them . One notable exception is a study by Cho and Garcia-Molina , who crawled a set of 720,000 pages on a daily basis over four months , and counted pages as having changed if their MD5 checksum changed . They found that 40 % of all web pages in their set changed within a week , and 23 % of those pages that fell into the . com domain changed daily . This paper expands on Cho and Garcia-Molina 's study , both in terms of coverage and in terms of sensitivity to change . We crawled a set of 150,836,209 HTML pages once every week , over a span of 11 weeks . For each page , we recorded a checksum of the page , and a feature vector of the words on the page , plus various other data such as the page length , the HTTP status code , etc. . Moreover , we pseudo-randomly selected 0.1 % of all of our URLs , and saved the full text of each download of the corresponding pages . After completion of the crawl , we analyzed the degree of change of each page , and investigated which factors are correlated with change intensity . We found that the average degree of change varies widely across top-level domains , and that larger pages change more often and more severely than smaller ones . This paper describes the crawl and the data transformations we performed on the logs , and presents some statistical observations on the degree of change of different classes of pages .
doc_bio_tags: O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O