title: Towards Visual Text Grounding of Multimodal Large Language Model
summary: Despite the existing evolution of Multimodal Large Language Models (MLLMs), a
  non-neglectable limitation remains in their struggle with visual text
  grounding, especially in text-rich images of documents. Document images, such
  as scanned forms and infographics, highlight critical challenges due to their
  complex layouts and textual content. However, current benchmarks do not fully
  address these challenges, as they mostly focus on visual grounding on natural
  images, rather than text-rich document images. Thus, to bridge this gap, we
  introduce TRIG, a novel task with a newly designed instruction dataset for
  benchmarking and improving the Text-Rich Image Grounding capabilities of MLLMs
  in document question-answering. Specifically, we propose an OCR-LLM-human
  interaction pipeline to create 800 manually annotated question-answer pairs as
  a benchmark and a large-scale training set of 90$ synthetic data based on four
  diverse datasets. A comprehensive evaluation of various MLLMs on our proposed
  benchmark exposes substantial limitations in their grounding capability on
  text-rich images. In addition, we propose two simple and effective TRIG methods
  based on general instruction tuning and plug-and-play efficient embedding,
  respectively. By finetuning MLLMs on our synthetic dataset, they promisingly
  improve spatial reasoning and grounding capabilities.
paper_authors._id: 67f8932aa1d82990423d99b5
paper_authors.name: Ming Li
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-11T03:57:35.193Z
paper_authors._id: 65031d01cccc7b28a388c719
paper_authors.avatarUrl: /avatars/9d8c94b6ab8ad8b4faba3221b7e76053.svg
paper_authors.fullname: Ming Li
paper_authors.type: user
paper_authors.user: MingLiiii
paper_authors._id: 67f8932aa1d82990423d99b6
paper_authors.name: Ruiyi Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-11T03:57:35.193Z
paper_authors._id: 64668f982da1abc242355cbb
paper_authors.avatarUrl: /avatars/c9a248b8d70b1a8f7b78265c98690570.svg
paper_authors.fullname: Ruiyi Zhang
paper_authors.type: user
paper_authors.user: zhangry868
paper_authors._id: 67f8932aa1d82990423d99b7
paper_authors.name: Jian Chen
paper_authors._id: 67f8932aa1d82990423d99b8
paper_authors.name: Jiuxiang Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:36:21.275Z
paper_authors._id: 642467708d97ce93878e8124
paper_authors.avatarUrl: /avatars/f0e464ddb4bd790f470fc0f10275fa26.svg
paper_authors.fullname: Jiuxiang Gu
paper_authors.type: user
paper_authors.user: JoshuaGu
paper_authors._id: 67f8932aa1d82990423d99b9
paper_authors.name: Yufan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:36:40.076Z
paper_authors._id: 635c2c2a7a165601151d3f85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
paper_authors.fullname: Yufan Zhou
paper_authors.type: user
paper_authors.user: YfZ
paper_authors._id: 67f8932aa1d82990423d99ba
paper_authors.name: Franck Dernoncourt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-11T07:25:03.265Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_authors._id: 67f8932aa1d82990423d99bb
paper_authors.name: Wanrong Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:36:47.002Z
paper_authors._id: 633a7e91c8eb7b089034fb3c
paper_authors.avatarUrl: /avatars/0e03d80aa469de131cc17bb0c5c016b9.svg
paper_authors.fullname: Wanrong Zhu
paper_authors.type: user
paper_authors.user: VegB
paper_authors._id: 67f8932aa1d82990423d99bc
paper_authors.name: Tianyi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T08:37:06.061Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_authors._id: 67f8932aa1d82990423d99bd
paper_authors.name: Tong Sun
paper_title: Towards Visual Text Grounding of Multimodal Large Language Model
paper_summary: Despite the existing evolution of Multimodal Large Language Models (MLLMs), a
  non-neglectable limitation remains in their struggle with visual text
  grounding, especially in text-rich images of documents. Document images, such
  as scanned forms and infographics, highlight critical challenges due to their
  complex layouts and textual content. However, current benchmarks do not fully
  address these challenges, as they mostly focus on visual grounding on natural
  images, rather than text-rich document images. Thus, to bridge this gap, we
  introduce TRIG, a novel task with a newly designed instruction dataset for
  benchmarking and improving the Text-Rich Image Grounding capabilities of MLLMs
  in document question-answering. Specifically, we propose an OCR-LLM-human
  interaction pipeline to create 800 manually annotated question-answer pairs as
  a benchmark and a large-scale training set of 90$ synthetic data based on four
  diverse datasets. A comprehensive evaluation of various MLLMs on our proposed
  benchmark exposes substantial limitations in their grounding capability on
  text-rich images. In addition, we propose two simple and effective TRIG methods
  based on general instruction tuning and plug-and-play efficient embedding,
  respectively. By finetuning MLLMs on our synthetic dataset, they promisingly
  improve spatial reasoning and grounding capabilities.
paper_ai_summary: A new task, TRIG, with an instruction dataset benchmarking MLLMs on text-rich document images improves grounding capabilities through OCR-LLM-human interaction and finetuning.
paper_ai_keywords: Multimodal Large Language Models MLMMs visual text grounding document images OCR-LLM-human interaction instruction dataset general instruction tuning plug-and-play efficient embedding spatial reasoning