title: SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
summary: Based on powerful Large Language Models (LLMs), recent generative Multimodal
  Large Language Models (MLLMs) have gained prominence as a pivotal research
  area, exhibiting remarkable capability for both comprehension and generation.
  In this work, we address the evaluation of generative comprehension in MLLMs as
  a preliminary step towards a comprehensive assessment of generative models, by
  introducing a benchmark named SEED-Bench. SEED-Bench consists of 19K multiple
  choice questions with accurate human annotations (x 6 larger than existing
  benchmarks), which spans 12 evaluation dimensions including the comprehension
  of both the image and video modality. We develop an advanced pipeline for
  generating multiple-choice questions that target specific evaluation
  dimensions, integrating both automatic filtering and manual verification
  processes. Multiple-choice questions with groundtruth options derived from
  human annotation enables an objective and efficient assessment of model
  performance, eliminating the need for human or GPT intervention during
  evaluation. We further evaluate the performance of 18 models across all 12
  dimensions, covering both the spatial and temporal understanding. By revealing
  the limitations of existing MLLMs through evaluation results, we aim for
  SEED-Bench to provide insights for motivating future research. We will launch
  and consistently maintain a leaderboard to provide a platform for the community
  to assess and investigate model capability.
paper_authors._id: 64c86785cb2f1bf0e7b7c2e8
paper_authors.name: Bohao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-21T09:56:43.623Z
paper_authors._id: 6310b7e70a43f97f6c56191e
paper_authors.avatarUrl: /avatars/4a24c76e34d12c3d6230a4a081115f72.svg
paper_authors.fullname: Bohao Li
paper_authors.type: user
paper_authors.user: BreakLee
paper_authors._id: 64c86785cb2f1bf0e7b7c2e9
paper_authors.name: Rui Wang
paper_authors._id: 64c86785cb2f1bf0e7b7c2ea
paper_authors.name: Guangzhi Wang
paper_authors._id: 64c86785cb2f1bf0e7b7c2eb
paper_authors.name: Yuying Ge
paper_authors._id: 64c86785cb2f1bf0e7b7c2ec
paper_authors.name: Yixiao Ge
paper_authors._id: 64c86785cb2f1bf0e7b7c2ed
paper_authors.name: Ying Shan
paper_title: SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
paper_summary: Based on powerful Large Language Models (LLMs), recent generative Multimodal
  Large Language Models (MLLMs) have gained prominence as a pivotal research
  area, exhibiting remarkable capability for both comprehension and generation.
  In this work, we address the evaluation of generative comprehension in MLLMs as
  a preliminary step towards a comprehensive assessment of generative models, by
  introducing a benchmark named SEED-Bench. SEED-Bench consists of 19K multiple
  choice questions with accurate human annotations (x 6 larger than existing
  benchmarks), which spans 12 evaluation dimensions including the comprehension
  of both the image and video modality. We develop an advanced pipeline for
  generating multiple-choice questions that target specific evaluation
  dimensions, integrating both automatic filtering and manual verification
  processes. Multiple-choice questions with groundtruth options derived from
  human annotation enables an objective and efficient assessment of model
  performance, eliminating the need for human or GPT intervention during
  evaluation. We further evaluate the performance of 18 models across all 12
  dimensions, covering both the spatial and temporal understanding. By revealing
  the limitations of existing MLLMs through evaluation results, we aim for
  SEED-Bench to provide insights for motivating future research. We will launch
  and consistently maintain a leaderboard to provide a platform for the community
  to assess and investigate model capability.
paper_ai_keywords: MLLMs SEED-Bench multiple choice questions human annotations evaluation dimensions automatic filtering manual verification groundtruth options model performance spatial understanding temporal understanding