title: SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with
    Text-Rich Visual Comprehension
summary: Comprehending text-rich visual content is paramount for the practical
  application of Multimodal Large Language Models (MLLMs), since text-rich
  scenarios are ubiquitous in the real world, which are characterized by the
  presence of extensive texts embedded within images. Recently, the advent of
  MLLMs with impressive versatility has raised the bar for what we can expect
  from MLLMs. However, their proficiency in text-rich scenarios has yet to be
  comprehensively and objectively assessed, since current MLLM benchmarks
  primarily focus on evaluating general visual comprehension. In this work, we
  introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating
  text-rich visual comprehension of MLLMs. Our benchmark comprises 2.3K
  multiple-choice questions with precise human annotations, spanning three broad
  categories: Charts, Maps, and Webs, each of which covers a wide spectrum of
  text-rich scenarios in the real world. These categories, due to their inherent
  complexity and diversity, effectively simulate real-world text-rich
  environments. We further conduct a thorough evaluation involving 34 prominent
  MLLMs (including GPT-4V, Gemini-Pro-Vision and Claude-3-Opus) and emphasize the
  current limitations of MLLMs in text-rich visual comprehension. We hope that
  our work can serve as a valuable addition to existing MLLM benchmarks,
  providing insightful observations and inspiring further research in the area of
  text-rich visual comprehension with MLLMs. The dataset and evaluation code can
  be accessed at https://github.com/AILab-CVC/SEED-Bench.
paper_authors._id: 662b12f3c90d81a396fa5b75
paper_authors.name: Bohao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:03:44.583Z
paper_authors._id: 6310b7e70a43f97f6c56191e
paper_authors.avatarUrl: /avatars/4a24c76e34d12c3d6230a4a081115f72.svg
paper_authors.fullname: Bohao Li
paper_authors.type: user
paper_authors.user: BreakLee
paper_authors._id: 662b12f3c90d81a396fa5b76
paper_authors.name: Yuying Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:03:56.492Z
paper_authors._id: 6455cc8f654d8bccae50e4d4
paper_authors.avatarUrl: /avatars/506a9992e5bf52e06d37cc22e4b307c0.svg
paper_authors.fullname: Yuying Ge
paper_authors.type: user
paper_authors.user: tttoaster
paper_authors._id: 662b12f3c90d81a396fa5b77
paper_authors.name: Yi Chen
paper_authors._id: 662b12f3c90d81a396fa5b78
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:04:23.679Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 662b12f3c90d81a396fa5b79
paper_authors.name: Ruimao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:04:35.029Z
paper_authors._id: 65324848220f490dbaa3b8c9
paper_authors.avatarUrl: /avatars/cec7dcadf4657bb4f61e2c9a8075521b.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Ruimao
paper_authors._id: 662b12f3c90d81a396fa5b7a
paper_authors.name: Ying Shan
paper_title: SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with
    Text-Rich Visual Comprehension
paper_summary: Comprehending text-rich visual content is paramount for the practical
  application of Multimodal Large Language Models (MLLMs), since text-rich
  scenarios are ubiquitous in the real world, which are characterized by the
  presence of extensive texts embedded within images. Recently, the advent of
  MLLMs with impressive versatility has raised the bar for what we can expect
  from MLLMs. However, their proficiency in text-rich scenarios has yet to be
  comprehensively and objectively assessed, since current MLLM benchmarks
  primarily focus on evaluating general visual comprehension. In this work, we
  introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating
  text-rich visual comprehension of MLLMs. Our benchmark comprises 2.3K
  multiple-choice questions with precise human annotations, spanning three broad
  categories: Charts, Maps, and Webs, each of which covers a wide spectrum of
  text-rich scenarios in the real world. These categories, due to their inherent
  complexity and diversity, effectively simulate real-world text-rich
  environments. We further conduct a thorough evaluation involving 34 prominent
  MLLMs (including GPT-4V, Gemini-Pro-Vision and Claude-3-Opus) and emphasize the
  current limitations of MLLMs in text-rich visual comprehension. We hope that
  our work can serve as a valuable addition to existing MLLM benchmarks,
  providing insightful observations and inspiring further research in the area of
  text-rich visual comprehension with MLLMs. The dataset and evaluation code can
  be accessed at https://github.com/AILab-CVC/SEED-Bench.
paper_ai_summary: A benchmark evaluating text-rich visual comprehension for Multimodal Large Language Models (MLLMs) introduces SEED-Bench-2-Plus with comprehensive human annotations across diverse scenarios.
paper_ai_keywords: Multimodal Large Language Models MLLMs text-rich visual comprehension SEED-Bench-2-Plus multiple-choice questions human annotations general visual comprehension GPT-4V Gemini-Pro-Vision Claude-3-Opus