ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in
  Large Multimodal Models
Recent advancements in AI have led to the development of large multimodal
models (LMMs) capable of processing complex tasks involving joint reasoning
over text and visual content in the image (e.g., navigating maps in public
places). This paper introduces ConTextual, a novel benchmark comprising
instructions designed explicitly to evaluate LMMs' ability to perform
context-sensitive text-rich visual reasoning. ConTextual emphasizes diverse
real-world scenarios (e.g., time-reading, navigation, shopping and more)
demanding a deeper understanding of the interactions between textual and visual
elements. Our findings reveal a significant performance gap of 30.8% between
the best-performing LMM, GPT-4V(ision), and human capabilities using human
evaluation indicating substantial room for improvement in context-sensitive
text-rich visual reasoning. Notably, while GPT-4V excelled in abstract
categories like meme and quote interpretation, its overall performance still
lagged behind humans. In addition to human evaluations, we also employed
automatic evaluation metrics using GPT-4, uncovering similar trends in
performance disparities. We also perform a fine-grained evaluation across
diverse visual contexts and provide qualitative analysis which provides a
robust framework for future advancements in the LMM design.
https://con-textual.github.io/
65b1e496ade89bc3fa6ad1d8
Rohan Wadhawan
claimed_verified
2024-01-25T08:04:01.991Z
63e099ddf351dc074575204a
/avatars/422234f3ceeb7f95fe41b25eb8442485.svg
Rohan Wadhawan
user
rohan598
65b1e496ade89bc3fa6ad1d9
Hritik Bansal
claimed_verified
2024-01-25T08:03:59.607Z
61c5c25705aa54027c52f7b3
/avatars/8a89e040dc331b7a83d9a704c4fc29d2.svg
Hritik Bansal
user
hbXNov
65b1e496ade89bc3fa6ad1da
Kai-Wei Chang
admin_assigned
2024-01-25T09:26:46.343Z
60b7b9d71b90c5d07c23fbd0
https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
Kai-Wei Chang
user
kaiweichang
65b1e496ade89bc3fa6ad1db
Nanyun Peng
admin_assigned
2024-01-25T09:27:08.572Z
64c13de54cbd12e168f09a58
https://cdn-avatars.huggingface.co/v1/production/uploads/64c13de54cbd12e168f09a58/Aw5BaNzf6uuwqQ333OQ5y.png
NANYUN (Violet) PENG
user
violetpeng
ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in
  Large Multimodal Models
Recent advancements in AI have led to the development of large multimodal
models (LMMs) capable of processing complex tasks involving joint reasoning
over text and visual content in the image (e.g., navigating maps in public
places). This paper introduces ConTextual, a novel benchmark comprising
instructions designed explicitly to evaluate LMMs' ability to perform
context-sensitive text-rich visual reasoning. ConTextual emphasizes diverse
real-world scenarios (e.g., time-reading, navigation, shopping and more)
demanding a deeper understanding of the interactions between textual and visual
elements. Our findings reveal a significant performance gap of 30.8% between
the best-performing LMM, GPT-4V(ision), and human capabilities using human
evaluation indicating substantial room for improvement in context-sensitive
text-rich visual reasoning. Notably, while GPT-4V excelled in abstract
categories like meme and quote interpretation, its overall performance still
lagged behind humans. In addition to human evaluations, we also employed
automatic evaluation metrics using GPT-4, uncovering similar trends in
performance disparities. We also perform a fine-grained evaluation across
diverse visual contexts and provide qualitative analysis which provides a
robust framework for future advancements in the LMM design.
https://con-textual.github.io/
A new benchmark, ConTextual, evaluates LMMs' context-sensitive text-rich visual reasoning, revealing a significant performance gap compared to humans.
multimodal models ConTextual text-rich visual reasoning GPT-4V human evaluation automatic evaluation metrics visual contexts qualitative analysis