title: M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding
    And A Retrieval-Aware Tuning Framework
summary: The ability to understand and answer questions over documents can be useful
  in many business and practical applications. However, documents often contain
  lengthy and diverse multimodal contents such as texts, figures, and tables,
  which are very time-consuming for humans to read thoroughly. Hence, there is an
  urgent need to develop effective and automated methods to aid humans in this
  task. In this work, we introduce M-LongDoc, a benchmark of 851 samples, and an
  automated framework to evaluate the performance of large multimodal models. We
  further propose a retrieval-aware tuning approach for efficient and effective
  multimodal document reading. Compared to existing works, our benchmark consists
  of more recent and lengthy documents with hundreds of pages, while also
  requiring open-ended solutions and not just extractive answers. To our
  knowledge, our training framework is the first to directly address the
  retrieval setting for multimodal long documents. To enable tuning open-source
  models, we construct a training corpus in a fully automatic manner for the
  question-answering task over such documents. Experiments show that our tuning
  approach achieves a relative improvement of 4.6% for the correctness of model
  responses, compared to the baseline open-source models. Our data, code, and
  models are available at https://multimodal-documents.github.io.
paper_authors._id: 6732c7907c79bb4d0668912f
paper_authors.name: Yew Ken Chia
paper_authors._id: 6732c7907c79bb4d06689130
paper_authors.name: Liying Cheng
paper_authors._id: 6732c7907c79bb4d06689131
paper_authors.name: Hou Pong Chan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-18T15:19:59.511Z
paper_authors._id: 604f67ef0fe8ff3ec13d71ef
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/604f67ef0fe8ff3ec13d71ef/KhUwWvZ3OJ9nEee3B-SXO.png
paper_authors.fullname: Hou Pong (Ken) Chan
paper_authors.type: user
paper_authors.user: kenchan0226
paper_authors._id: 6732c7907c79bb4d06689132
paper_authors.name: Chaoqun Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-06T09:28:08.556Z
paper_authors._id: 61657b0b20606e5e73f611cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61657b0b20606e5e73f611cc/6ZPne2GYlWkxrx35ND1P8.png
paper_authors.fullname: CHAOQUN LIU
paper_authors.type: user
paper_authors.user: lukecq
paper_authors._id: 6732c7907c79bb4d06689133
paper_authors.name: Maojia Song
paper_authors._id: 6732c7907c79bb4d06689134
paper_authors.name: Sharifah Mahani Aljunied
paper_authors._id: 6732c7907c79bb4d06689135
paper_authors.name: Soujanya Poria
paper_authors._id: 6732c7907c79bb4d06689136
paper_authors.name: Lidong Bing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:03:21.345Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_title: M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding
    And A Retrieval-Aware Tuning Framework
paper_summary: The ability to understand and answer questions over documents can be useful
  in many business and practical applications. However, documents often contain
  lengthy and diverse multimodal contents such as texts, figures, and tables,
  which are very time-consuming for humans to read thoroughly. Hence, there is an
  urgent need to develop effective and automated methods to aid humans in this
  task. In this work, we introduce M-LongDoc, a benchmark of 851 samples, and an
  automated framework to evaluate the performance of large multimodal models. We
  further propose a retrieval-aware tuning approach for efficient and effective
  multimodal document reading. Compared to existing works, our benchmark consists
  of more recent and lengthy documents with hundreds of pages, while also
  requiring open-ended solutions and not just extractive answers. To our
  knowledge, our training framework is the first to directly address the
  retrieval setting for multimodal long documents. To enable tuning open-source
  models, we construct a training corpus in a fully automatic manner for the
  question-answering task over such documents. Experiments show that our tuning
  approach achieves a relative improvement of 4.6% for the correctness of model
  responses, compared to the baseline open-source models. Our data, code, and
  models are available at https://multimodal-documents.github.io.
paper_ai_keywords: M-LongDoc benchmark automated framework retrieval-aware tuning multimodal document reading retrieval setting open-ended solutions extractive answers training corpus question-answering task correctness of model responses