The Availability and Persistence of Web References in D-Lib Magazine
<|reference_start|>The Availability and Persistence of Web References in D-Lib Magazine: We explore the availability and persistence of URLs cited in articles published in D-Lib Magazine. We extracted 4387 unique URLs referenced in 453 articles published from July 1995 to August 2004. The availability was checked three times a week for 25 weeks from September 2004 to February 2005. We found that approximately 28% of those URLs failed to resolve initially, and 30% failed to resolve at the last check. A majority of the unresolved URLs were due to 404 (page not found) and 500 (internal server error) errors. The content pointed to by the URLs was relatively stable; only 16% of the content registered more than a 1 KB change during the testing period. We explore possible factors which may cause a URL to fail by examining its age, path depth, top-level domain and file extension. Based on the data collected, we found the half-life of a URL referenced in a D-Lib Magazine article is approximately 10 years. We also found that URLs were more likely to be unavailable if they pointed to resources in the .net, .edu or country-specific top-level domain, used non-standard ports (i.e., not port 80), or pointed to resources with uncommon or deprecated extensions (e.g., .shtml, .ps, .txt).<|reference_end|>
@article{mccown2005the,
    title={The Availability and Persistence of Web References in D-Lib Magazine},
    author={Frank McCown, Sheffan Chan, Michael L. Nelson, Johan Bollen},
    journal={5th International Web Archiving Workshop (IWAW05), Vienna,
  Austria, 2005},
    year={2005},
    archivePrefix={arXiv},
    eprint={cs/0511077},
    primaryClass={cs.DL}
}