title: LoRA-Contextualizing Adaptation of Large Multimodal Models for Long
    Document Understanding
summary: Large multimodal models (LMMs) have recently shown great progress in
  text-rich image understanding, yet they still struggle with complex,
  multi-page, visually-rich documents. Traditional methods using document parsers
  for retrieval-augmented generation suffer from performance and efficiency
  limitations, while directly presenting all pages to LMMs leads to
  inefficiencies, especially with lengthy documents. In this work, we present a
  novel framework named LoRA-Contextualizing Adaptation of Large multimodal
  models (LoCAL), which broadens the capabilities of any LMM to support
  long-document understanding. We demonstrate that LMMs can effectively serve as
  multimodal retrievers, fetching relevant pages to answer user questions based
  on these pages. LoCAL is implemented with two specific LMM adapters: one for
  evidence page retrieval and another for question answering. Empirical results
  show state-of-the-art performance on public benchmarks, demonstrating the
  effectiveness of LoCAL.
paper_authors._id: 672a5ab6aa84596b8ab0b88d
paper_authors.name: Jian Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:37:37.636Z
paper_authors._id: 64593836c16ecb4815e082ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64593836c16ecb4815e082ce/T9WM0Ly6ADi3a9JUSIsSV.png
paper_authors.fullname: Jian Chen
paper_authors.type: user
paper_authors.user: puar-playground
paper_authors._id: 672a5ab6aa84596b8ab0b88e
paper_authors.name: Ruiyi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T20:17:19.577Z
paper_authors._id: 64668f982da1abc242355cbb
paper_authors.avatarUrl: /avatars/c9a248b8d70b1a8f7b78265c98690570.svg
paper_authors.fullname: Ruiyi Zhang
paper_authors.type: user
paper_authors.user: zhangry868
paper_authors._id: 672a5ab6aa84596b8ab0b88f
paper_authors.name: Yufan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:05:08.902Z
paper_authors._id: 635c2c2a7a165601151d3f85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
paper_authors.fullname: Yufan Zhou
paper_authors.type: user
paper_authors.user: YfZ
paper_authors._id: 672a5ab6aa84596b8ab0b890
paper_authors.name: Tong Yu
paper_authors._id: 672a5ab6aa84596b8ab0b891
paper_authors.name: Franck Dernoncourt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T20:17:25.830Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_authors._id: 672a5ab6aa84596b8ab0b892
paper_authors.name: Jiuxiang Gu
paper_authors._id: 672a5ab6aa84596b8ab0b893
paper_authors.name: Ryan A. Rossi
paper_authors._id: 672a5ab6aa84596b8ab0b894
paper_authors.name: Changyou Chen
paper_authors._id: 672a5ab6aa84596b8ab0b895
paper_authors.name: Tong Sun
paper_title: LoRA-Contextualizing Adaptation of Large Multimodal Models for Long
    Document Understanding
paper_summary: Large multimodal models (LMMs) have recently shown great progress in
  text-rich image understanding, yet they still struggle with complex,
  multi-page, visually-rich documents. Traditional methods using document parsers
  for retrieval-augmented generation suffer from performance and efficiency
  limitations, while directly presenting all pages to LMMs leads to
  inefficiencies, especially with lengthy documents. In this work, we present a
  novel framework named LoRA-Contextualizing Adaptation of Large multimodal
  models (LoCAL), which broadens the capabilities of any LMM to support
  long-document understanding. We demonstrate that LMMs can effectively serve as
  multimodal retrievers, fetching relevant pages to answer user questions based
  on these pages. LoCAL is implemented with two specific LMM adapters: one for
  evidence page retrieval and another for question answering. Empirical results
  show state-of-the-art performance on public benchmarks, demonstrating the
  effectiveness of LoCAL.
paper_ai_keywords: multimodal models (LMMs) text-rich image understanding multi-page, visually-rich documents document parsers retrieval-augmented generation inefficiencies long-document understanding LoRA-Contextualizing Adaptation of Large multimodal models (LoCAL) multimodal retrievers evidence page retrieval question answering state-of-the-art benchmarks