title: CRAWLDoc: A Dataset for Robust Ranking of Bibliographic Documents
summary: Publication databases rely on accurate metadata extraction from diverse web
  sources, yet variations in web layouts and data formats present challenges for
  metadata providers. This paper introduces CRAWLDoc, a new method for contextual
  ranking of linked web documents. Starting with a publication's URL, such as a
  digital object identifier, CRAWLDoc retrieves the landing page and all linked
  web resources, including PDFs, ORCID profiles, and supplementary materials. It
  embeds these resources, along with anchor texts and the URLs, into a unified
  representation. For evaluating CRAWLDoc, we have created a new, manually
  labeled dataset of 600 publications from six top publishers in computer
  science. Our method CRAWLDoc demonstrates a robust and layout-independent
  ranking of relevant documents across publishers and data formats. It lays the
  foundation for improved metadata extraction from web documents with various
  layouts and formats. Our source code and dataset can be accessed at
  https://github.com/FKarl/CRAWLDoc.
paper_authors._id: 6841be30ea0023757b685a3b
paper_authors.name: Fabian Karl
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T17:07:58.514Z
paper_authors._id: 636c1de8cfb49b468218e21e
paper_authors.avatarUrl: /avatars/ab7d7b4b7c1f0a9b2a9625f292d0a6a5.svg
paper_authors.fullname: Fabian Karl
paper_authors.type: user
paper_authors.user: FabianKarl
paper_authors._id: 6841be30ea0023757b685a3c
paper_authors.name: Ansgar Scherp
paper_title: CRAWLDoc: A Dataset for Robust Ranking of Bibliographic Documents
paper_summary: Publication databases rely on accurate metadata extraction from diverse web
  sources, yet variations in web layouts and data formats present challenges for
  metadata providers. This paper introduces CRAWLDoc, a new method for contextual
  ranking of linked web documents. Starting with a publication's URL, such as a
  digital object identifier, CRAWLDoc retrieves the landing page and all linked
  web resources, including PDFs, ORCID profiles, and supplementary materials. It
  embeds these resources, along with anchor texts and the URLs, into a unified
  representation. For evaluating CRAWLDoc, we have created a new, manually
  labeled dataset of 600 publications from six top publishers in computer
  science. Our method CRAWLDoc demonstrates a robust and layout-independent
  ranking of relevant documents across publishers and data formats. It lays the
  foundation for improved metadata extraction from web documents with various
  layouts and formats. Our source code and dataset can be accessed at
  https://github.com/FKarl/CRAWLDoc.
paper_ai_summary: CRAWLDoc is a method for ranking linked web documents by embedding resources and metadata into a unified representation, enabling robust metadata extraction across diverse web layouts and formats.
paper_ai_keywords: contextual ranking linked web documents CRAWLDoc landing page PDFs ORCID profiles supplementary materials unified representation anchor texts URL metadata extraction layout-independent ranking