M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding
  And A Retrieval-Aware Tuning Framework
The ability to understand and answer questions over documents can be useful
in many business and practical applications. However, documents often contain
lengthy and diverse multimodal contents such as texts, figures, and tables,
which are very time-consuming for humans to read thoroughly. Hence, there is an
urgent need to develop effective and automated methods to aid humans in this
task. In this work, we introduce M-LongDoc, a benchmark of 851 samples, and an
automated framework to evaluate the performance of large multimodal models. We
further propose a retrieval-aware tuning approach for efficient and effective
multimodal document reading. Compared to existing works, our benchmark consists
of more recent and lengthy documents with hundreds of pages, while also
requiring open-ended solutions and not just extractive answers. To our
knowledge, our training framework is the first to directly address the
retrieval setting for multimodal long documents. To enable tuning open-source
models, we construct a training corpus in a fully automatic manner for the
question-answering task over such documents. Experiments show that our tuning
approach achieves a relative improvement of 4.6% for the correctness of model
responses, compared to the baseline open-source models. Our data, code, and
models are available at https://multimodal-documents.github.io.
6732c7907c79bb4d0668912f
Yew Ken Chia
6732c7907c79bb4d06689130
Liying Cheng
6732c7907c79bb4d06689131
Hou Pong Chan
claimed_verified
2024-11-18T15:19:59.511Z
604f67ef0fe8ff3ec13d71ef
https://cdn-avatars.huggingface.co/v1/production/uploads/604f67ef0fe8ff3ec13d71ef/KhUwWvZ3OJ9nEee3B-SXO.png
Hou Pong (Ken) Chan
user
kenchan0226
6732c7907c79bb4d06689132
Chaoqun Liu
claimed_verified
2025-03-06T09:28:08.556Z
61657b0b20606e5e73f611cc
https://cdn-avatars.huggingface.co/v1/production/uploads/61657b0b20606e5e73f611cc/6ZPne2GYlWkxrx35ND1P8.png
CHAOQUN LIU
user
lukecq
6732c7907c79bb4d06689133
Maojia Song
6732c7907c79bb4d06689134
Sharifah Mahani Aljunied
6732c7907c79bb4d06689135
Soujanya Poria
6732c7907c79bb4d06689136
Lidong Bing
claimed_verified
2025-01-05T23:03:21.345Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding
  And A Retrieval-Aware Tuning Framework
The ability to understand and answer questions over documents can be useful
in many business and practical applications. However, documents often contain
lengthy and diverse multimodal contents such as texts, figures, and tables,
which are very time-consuming for humans to read thoroughly. Hence, there is an
urgent need to develop effective and automated methods to aid humans in this
task. In this work, we introduce M-LongDoc, a benchmark of 851 samples, and an
automated framework to evaluate the performance of large multimodal models. We
further propose a retrieval-aware tuning approach for efficient and effective
multimodal document reading. Compared to existing works, our benchmark consists
of more recent and lengthy documents with hundreds of pages, while also
requiring open-ended solutions and not just extractive answers. To our
knowledge, our training framework is the first to directly address the
retrieval setting for multimodal long documents. To enable tuning open-source
models, we construct a training corpus in a fully automatic manner for the
question-answering task over such documents. Experiments show that our tuning
approach achieves a relative improvement of 4.6% for the correctness of model
responses, compared to the baseline open-source models. Our data, code, and
models are available at https://multimodal-documents.github.io.
A benchmark and framework named M-LongDoc evaluates and improves large multimodal models for document reading, focusing on open-ended solutions and retrieval tasks over lengthy documents.
multimodal models M-LongDoc benchmark automated framework retrieval-aware tuning question-answering task