Towards Visual Text Grounding of Multimodal Large Language Model
Despite the existing evolution of Multimodal Large Language Models (MLLMs), a
non-neglectable limitation remains in their struggle with visual text
grounding, especially in text-rich images of documents. Document images, such
as scanned forms and infographics, highlight critical challenges due to their
complex layouts and textual content. However, current benchmarks do not fully
address these challenges, as they mostly focus on visual grounding on natural
images, rather than text-rich document images. Thus, to bridge this gap, we
introduce TRIG, a novel task with a newly designed instruction dataset for
benchmarking and improving the Text-Rich Image Grounding capabilities of MLLMs
in document question-answering. Specifically, we propose an OCR-LLM-human
interaction pipeline to create 800 manually annotated question-answer pairs as
a benchmark and a large-scale training set of 90$ synthetic data based on four
diverse datasets. A comprehensive evaluation of various MLLMs on our proposed
benchmark exposes substantial limitations in their grounding capability on
text-rich images. In addition, we propose two simple and effective TRIG methods
based on general instruction tuning and plug-and-play efficient embedding,
respectively. By finetuning MLLMs on our synthetic dataset, they promisingly
improve spatial reasoning and grounding capabilities.
67f8932aa1d82990423d99b5
Ming Li
extracted_pending
2025-04-11T03:57:35.193Z
65031d01cccc7b28a388c719
/avatars/9d8c94b6ab8ad8b4faba3221b7e76053.svg
Ming Li
user
MingLiiii
67f8932aa1d82990423d99b6
Ruiyi Zhang
extracted_pending
2025-04-11T03:57:35.193Z
64668f982da1abc242355cbb
/avatars/c9a248b8d70b1a8f7b78265c98690570.svg
Ruiyi Zhang
user
zhangry868
67f8932aa1d82990423d99b7
Jian Chen
claimed_verified
2025-07-03T07:43:25.804Z
64593836c16ecb4815e082ce
https://cdn-avatars.huggingface.co/v1/production/uploads/64593836c16ecb4815e082ce/T9WM0Ly6ADi3a9JUSIsSV.png
Jian Chen
user
puar-playground
67f8932aa1d82990423d99b8
Jiuxiang Gu
admin_assigned
2025-04-11T08:36:21.275Z
642467708d97ce93878e8124
/avatars/f0e464ddb4bd790f470fc0f10275fa26.svg
Jiuxiang Gu
user
JoshuaGu
67f8932aa1d82990423d99b9
Yufan Zhou
admin_assigned
2025-04-11T08:36:40.076Z
635c2c2a7a165601151d3f85
https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
Yufan Zhou
user
YfZ
67f8932aa1d82990423d99ba
Franck Dernoncourt
claimed_verified
2025-04-11T07:25:03.265Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
67f8932aa1d82990423d99bb
Wanrong Zhu
admin_assigned
2025-04-11T08:36:47.002Z
633a7e91c8eb7b089034fb3c
/avatars/0e03d80aa469de131cc17bb0c5c016b9.svg
Wanrong Zhu
user
VegB
67f8932aa1d82990423d99bc
Tianyi Zhou
admin_assigned
2025-04-11T08:37:06.061Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
67f8932aa1d82990423d99bd
Tong Sun
Towards Visual Text Grounding of Multimodal Large Language Model
Despite the existing evolution of Multimodal Large Language Models (MLLMs), a
non-neglectable limitation remains in their struggle with visual text
grounding, especially in text-rich images of documents. Document images, such
as scanned forms and infographics, highlight critical challenges due to their
complex layouts and textual content. However, current benchmarks do not fully
address these challenges, as they mostly focus on visual grounding on natural
images, rather than text-rich document images. Thus, to bridge this gap, we
introduce TRIG, a novel task with a newly designed instruction dataset for
benchmarking and improving the Text-Rich Image Grounding capabilities of MLLMs
in document question-answering. Specifically, we propose an OCR-LLM-human
interaction pipeline to create 800 manually annotated question-answer pairs as
a benchmark and a large-scale training set of 90$ synthetic data based on four
diverse datasets. A comprehensive evaluation of various MLLMs on our proposed
benchmark exposes substantial limitations in their grounding capability on
text-rich images. In addition, we propose two simple and effective TRIG methods
based on general instruction tuning and plug-and-play efficient embedding,
respectively. By finetuning MLLMs on our synthetic dataset, they promisingly
improve spatial reasoning and grounding capabilities.
A new task, TRIG, with an instruction dataset benchmarking MLLMs on text-rich document images improves grounding capabilities through OCR-LLM-human interaction and finetuning.
Multimodal Large Language Models MLMMs visual text grounding document images OCR-LLM-human interaction instruction dataset general instruction tuning plug-and-play efficient embedding spatial reasoning