title: ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in
    Large Multimodal Models
summary: Recent advancements in AI have led to the development of large multimodal
  models (LMMs) capable of processing complex tasks involving joint reasoning
  over text and visual content in the image (e.g., navigating maps in public
  places). This paper introduces ConTextual, a novel benchmark comprising
  instructions designed explicitly to evaluate LMMs' ability to perform
  context-sensitive text-rich visual reasoning. ConTextual emphasizes diverse
  real-world scenarios (e.g., time-reading, navigation, shopping and more)
  demanding a deeper understanding of the interactions between textual and visual
  elements. Our findings reveal a significant performance gap of 30.8% between
  the best-performing LMM, GPT-4V(ision), and human capabilities using human
  evaluation indicating substantial room for improvement in context-sensitive
  text-rich visual reasoning. Notably, while GPT-4V excelled in abstract
  categories like meme and quote interpretation, its overall performance still
  lagged behind humans. In addition to human evaluations, we also employed
  automatic evaluation metrics using GPT-4, uncovering similar trends in
  performance disparities. We also perform a fine-grained evaluation across
  diverse visual contexts and provide qualitative analysis which provides a
  robust framework for future advancements in the LMM design.
  https://con-textual.github.io/
paper_authors._id: 65b1e496ade89bc3fa6ad1d8
paper_authors.name: Rohan Wadhawan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-25T08:04:01.991Z
paper_authors._id: 63e099ddf351dc074575204a
paper_authors.avatarUrl: /avatars/422234f3ceeb7f95fe41b25eb8442485.svg
paper_authors.fullname: Rohan Wadhawan
paper_authors.type: user
paper_authors.user: rohan598
paper_authors._id: 65b1e496ade89bc3fa6ad1d9
paper_authors.name: Hritik Bansal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-25T08:03:59.607Z
paper_authors._id: 61c5c25705aa54027c52f7b3
paper_authors.avatarUrl: /avatars/8a89e040dc331b7a83d9a704c4fc29d2.svg
paper_authors.fullname: Hritik Bansal
paper_authors.type: user
paper_authors.user: hbXNov
paper_authors._id: 65b1e496ade89bc3fa6ad1da
paper_authors.name: Kai-Wei Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-25T09:26:46.343Z
paper_authors._id: 60b7b9d71b90c5d07c23fbd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
paper_authors.fullname: Kai-Wei Chang
paper_authors.type: user
paper_authors.user: kaiweichang
paper_authors._id: 65b1e496ade89bc3fa6ad1db
paper_authors.name: Nanyun Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-25T09:27:08.572Z
paper_authors._id: 64c13de54cbd12e168f09a58
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c13de54cbd12e168f09a58/Aw5BaNzf6uuwqQ333OQ5y.png
paper_authors.fullname: NANYUN (Violet) PENG
paper_authors.type: user
paper_authors.user: violetpeng
paper_title: ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in
    Large Multimodal Models
paper_summary: Recent advancements in AI have led to the development of large multimodal
  models (LMMs) capable of processing complex tasks involving joint reasoning
  over text and visual content in the image (e.g., navigating maps in public
  places). This paper introduces ConTextual, a novel benchmark comprising
  instructions designed explicitly to evaluate LMMs' ability to perform
  context-sensitive text-rich visual reasoning. ConTextual emphasizes diverse
  real-world scenarios (e.g., time-reading, navigation, shopping and more)
  demanding a deeper understanding of the interactions between textual and visual
  elements. Our findings reveal a significant performance gap of 30.8% between
  the best-performing LMM, GPT-4V(ision), and human capabilities using human
  evaluation indicating substantial room for improvement in context-sensitive
  text-rich visual reasoning. Notably, while GPT-4V excelled in abstract
  categories like meme and quote interpretation, its overall performance still
  lagged behind humans. In addition to human evaluations, we also employed
  automatic evaluation metrics using GPT-4, uncovering similar trends in
  performance disparities. We also perform a fine-grained evaluation across
  diverse visual contexts and provide qualitative analysis which provides a
  robust framework for future advancements in the LMM design.
  https://con-textual.github.io/
paper_ai_keywords: multimodal models (LMMs) joint reasoning text-rich visual reasoning ConTextual context-sensitive real-world scenarios time-reading navigation shopping human evaluation GPT-4V(ision) abstract categories meme interpretation quote interpretation automatic evaluation metrics fine-grained evaluation qualitative analysis