LoRA-Contextualizing Adaptation of Large Multimodal Models for Long
  Document Understanding
Large multimodal models (LMMs) have recently shown great progress in
text-rich image understanding, yet they still struggle with complex,
multi-page, visually-rich documents. Traditional methods using document parsers
for retrieval-augmented generation suffer from performance and efficiency
limitations, while directly presenting all pages to LMMs leads to
inefficiencies, especially with lengthy documents. In this work, we present a
novel framework named LoRA-Contextualizing Adaptation of Large multimodal
models (LoCAL), which broadens the capabilities of any LMM to support
long-document understanding. We demonstrate that LMMs can effectively serve as
multimodal retrievers, fetching relevant pages to answer user questions based
on these pages. LoCAL is implemented with two specific LMM adapters: one for
evidence page retrieval and another for question answering. Empirical results
show state-of-the-art performance on public benchmarks, demonstrating the
effectiveness of LoCAL.
672a5ab6aa84596b8ab0b88d
Jian Chen
claimed_verified
2024-11-15T09:37:37.636Z
64593836c16ecb4815e082ce
https://cdn-avatars.huggingface.co/v1/production/uploads/64593836c16ecb4815e082ce/T9WM0Ly6ADi3a9JUSIsSV.png
Jian Chen
user
puar-playground
672a5ab6aa84596b8ab0b88e
Ruiyi Zhang
claimed_verified
2024-11-05T20:17:19.577Z
64668f982da1abc242355cbb
/avatars/c9a248b8d70b1a8f7b78265c98690570.svg
Ruiyi Zhang
user
zhangry868
672a5ab6aa84596b8ab0b88f
Yufan Zhou
claimed_verified
2024-12-19T18:05:08.902Z
635c2c2a7a165601151d3f85
https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
Yufan Zhou
user
YfZ
672a5ab6aa84596b8ab0b890
Tong Yu
672a5ab6aa84596b8ab0b891
Franck Dernoncourt
claimed_verified
2024-11-05T20:17:25.830Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
672a5ab6aa84596b8ab0b892
Jiuxiang Gu
672a5ab6aa84596b8ab0b893
Ryan A. Rossi
672a5ab6aa84596b8ab0b894
Changyou Chen
672a5ab6aa84596b8ab0b895
Tong Sun
LoRA-Contextualizing Adaptation of Large Multimodal Models for Long
  Document Understanding
Large multimodal models (LMMs) have recently shown great progress in
text-rich image understanding, yet they still struggle with complex,
multi-page, visually-rich documents. Traditional methods using document parsers
for retrieval-augmented generation suffer from performance and efficiency
limitations, while directly presenting all pages to LMMs leads to
inefficiencies, especially with lengthy documents. In this work, we present a
novel framework named LoRA-Contextualizing Adaptation of Large multimodal
models (LoCAL), which broadens the capabilities of any LMM to support
long-document understanding. We demonstrate that LMMs can effectively serve as
multimodal retrievers, fetching relevant pages to answer user questions based
on these pages. LoCAL is implemented with two specific LMM adapters: one for
evidence page retrieval and another for question answering. Empirical results
show state-of-the-art performance on public benchmarks, demonstrating the
effectiveness of LoCAL.
LoCAL, a novel framework forLoRA-Contextualizing Adaptation, enhances large multimodal models' capability to understand and retrieve information from complex, multi-page documents.
LoRA-Contextualizing Adaptation large multimodal models LMMs evidence page retrieval question answering multimodal retrievers