title: OmniDocBench: Benchmarking Diverse PDF Document Parsing with
    Comprehensive Annotations
summary: Document content extraction is crucial in computer vision, especially for
  meeting the high-quality data needs of large language models (LLMs) and
  retrieval-augmented generation (RAG) technologies. However, current document
  parsing methods suffer from significant limitations in terms of diversity and
  comprehensive evaluation. To address these challenges, we introduce
  OmniDocBench, a novel multi-source benchmark designed to advance automated
  document content extraction. OmniDocBench includes a meticulously curated and
  annotated high-quality evaluation dataset comprising nine diverse document
  types, such as academic papers, textbooks, slides, among others. Our benchmark
  provides a flexible and comprehensive evaluation framework with 19 layout
  category labels and 14 attribute labels, enabling multi-level assessments
  across entire datasets, individual modules, or specific data types. Using
  OmniDocBench, we perform an exhaustive comparative analysis of existing modular
  pipelines and multimodal end-to-end methods, highlighting their limitations in
  handling document diversity and ensuring fair evaluation. OmniDocBench
  establishes a robust, diverse, and fair evaluation standard for the document
  content extraction field, offering crucial insights for future advancements and
  fostering the development of document parsing technologies. The codes and
  dataset is available in https://github.com/opendatalab/OmniDocBench.
paper_authors._id: 6758f2c55cd36ca4c1a4539b
paper_authors.name: Linke Ouyang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:20:06.000Z
paper_authors._id: 66753c556f2ac48ee625d7d1
paper_authors.avatarUrl: /avatars/8f7c252675fd8a096794d12971903722.svg
paper_authors.fullname: Linke Ouyang
paper_authors.type: user
paper_authors.user: ouyanglinke
paper_authors._id: 6758f2c55cd36ca4c1a4539c
paper_authors.name: Yuan Qu
paper_authors._id: 6758f2c55cd36ca4c1a4539d
paper_authors.name: Hongbin Zhou
paper_authors._id: 6758f2c55cd36ca4c1a4539e
paper_authors.name: Jiawei Zhu
paper_authors._id: 6758f2c55cd36ca4c1a4539f
paper_authors.name: Rui Zhang
paper_authors._id: 6758f2c55cd36ca4c1a453a0
paper_authors.name: Qunshu Lin
paper_authors._id: 6758f2c55cd36ca4c1a453a1
paper_authors.name: Bin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:20:11.073Z
paper_authors._id: 63ae9ff5557befe297a76f90
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1672388558183-noauth.jpeg
paper_authors.fullname: Bin Wang
paper_authors.type: user
paper_authors.user: wanderkid
paper_authors._id: 6758f2c55cd36ca4c1a453a2
paper_authors.name: Zhiyuan Zhao
paper_authors._id: 6758f2c55cd36ca4c1a453a3
paper_authors.name: Man Jiang
paper_authors._id: 6758f2c55cd36ca4c1a453a4
paper_authors.name: Xiaomeng Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-12T13:41:08.395Z
paper_authors._id: 668f77a3b3991ac0c308a441
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/668f77a3b3991ac0c308a441/w5-9UAxi32HnrTjW9_fGC.jpeg
paper_authors.fullname: xiaomeng zhao
paper_authors.type: user
paper_authors.user: myhloli
paper_authors._id: 6758f2c55cd36ca4c1a453a5
paper_authors.name: Jin Shi
paper_authors._id: 6758f2c55cd36ca4c1a453a6
paper_authors.name: Fan Wu
paper_authors._id: 6758f2c55cd36ca4c1a453a7
paper_authors.name: Pei Chu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:04:27.537Z
paper_authors._id: 64c9beb2904317f42de06dd8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c9beb2904317f42de06dd8/he3rxfyzfwEd1vLuK6_o2.jpeg
paper_authors.fullname: Pei Chu
paper_authors.type: user
paper_authors.user: chupei
paper_authors._id: 6758f2c55cd36ca4c1a453a8
paper_authors.name: Minghao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:20:08.296Z
paper_authors._id: 6417d9ea8f689506e7148417
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
paper_authors.fullname: minghao
paper_authors.type: user
paper_authors.user: Liam-Liu
paper_authors._id: 6758f2c55cd36ca4c1a453a9
paper_authors.name: Zhenxiang Li
paper_authors._id: 6758f2c55cd36ca4c1a453aa
paper_authors.name: Chao Xu
paper_authors._id: 6758f2c55cd36ca4c1a453ab
paper_authors.name: Bo Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T09:53:47.544Z
paper_authors._id: 643dfd235aafbdca3a5792c0
paper_authors.avatarUrl: /avatars/ce8553cf5936012c692e08054ee27937.svg
paper_authors.fullname: Bo Zhang
paper_authors.type: user
paper_authors.user: BoZhang
paper_authors._id: 6758f2c55cd36ca4c1a453ac
paper_authors.name: Botian Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:17:19.807Z
paper_authors._id: 643df87f7cd64d872cb9fabd
paper_authors.avatarUrl: /avatars/c53bfabcee08de448dde973915e8b31d.svg
paper_authors.fullname: Botian Shi
paper_authors.type: user
paper_authors.user: friskit
paper_authors._id: 6758f2c55cd36ca4c1a453ad
paper_authors.name: Zhongying Tu
paper_authors._id: 6758f2c55cd36ca4c1a453ae
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:17:05.536Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_title: OmniDocBench: Benchmarking Diverse PDF Document Parsing with
    Comprehensive Annotations
paper_summary: Document content extraction is crucial in computer vision, especially for
  meeting the high-quality data needs of large language models (LLMs) and
  retrieval-augmented generation (RAG) technologies. However, current document
  parsing methods suffer from significant limitations in terms of diversity and
  comprehensive evaluation. To address these challenges, we introduce
  OmniDocBench, a novel multi-source benchmark designed to advance automated
  document content extraction. OmniDocBench includes a meticulously curated and
  annotated high-quality evaluation dataset comprising nine diverse document
  types, such as academic papers, textbooks, slides, among others. Our benchmark
  provides a flexible and comprehensive evaluation framework with 19 layout
  category labels and 14 attribute labels, enabling multi-level assessments
  across entire datasets, individual modules, or specific data types. Using
  OmniDocBench, we perform an exhaustive comparative analysis of existing modular
  pipelines and multimodal end-to-end methods, highlighting their limitations in
  handling document diversity and ensuring fair evaluation. OmniDocBench
  establishes a robust, diverse, and fair evaluation standard for the document
  content extraction field, offering crucial insights for future advancements and
  fostering the development of document parsing technologies. The codes and
  dataset is available in https://github.com/opendatalab/OmniDocBench.