Dennis Fetterly, Mark Manasse, Marc Najork, and Janet Wiener How
fast does the web change? Does most of the content remain
unchanged once it has been authored, or are the documents
continuously updated? Do pages change a little or a lot? Is the
extent of change correlated to any other property of the page?
All of these questions are of interest to those who mine the
web, including all the popular search engines, but few studies
have been performed to date to answer them. One notable
exception is a study by Cho and Garcia-Molina, who crawled a set
of 720,000 pages on a daily basis over four months, and counted
pages as having changed if their MD5 checksum changed. They
found that 40% of all web pages in their set changed within a
week, and 23% of those pages that fell into the .com domain
changed daily. This paper expands on Cho and Garcia-Molina’s
study, both in terms of coverage and in terms of sensitivity to
change. We crawled a set of 150,836,209 HTML pages once every
week, over a span of 11 weeks. For eac