title: CapArena: Benchmarking and Analyzing Detailed Image Captioning in the
    LLM Era
summary: Image captioning has been a longstanding challenge in vision-language
  research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate
  detailed and comprehensive image descriptions. However, benchmarking the
  quality of such captions remains unresolved. This paper addresses two key
  questions: (1) How well do current VLMs actually perform on image captioning,
  particularly compared to humans? We built CapArena, a platform with over 6000
  pairwise caption battles and high-quality human preference votes. Our
  arena-style evaluation marks a milestone, showing that leading models like
  GPT-4o achieve or even surpass human performance, while most open-source models
  lag behind. (2) Can automated metrics reliably assess detailed caption quality?
  Using human annotations from CapArena, we evaluate traditional and recent
  captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while
  some metrics (e.g., METEOR) show decent caption-level agreement with humans,
  their systematic biases lead to inconsistencies in model ranking. In contrast,
  VLM-as-a-Judge demonstrates robust discernment at both the caption and model
  levels. Building on these insights, we release CapArena-Auto, an accurate and
  efficient automated benchmark for detailed captioning, achieving 94.3%
  correlation with human rankings at just $4 per test. Data and resources will be
  open-sourced at https://caparena.github.io.
paper_authors._id: 67d8e115f55b855ae6d8f29b
paper_authors.name: Kanzhi Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T11:34:32.729Z
paper_authors._id: 63340dbbd92c5842ae71d1e9
paper_authors.avatarUrl: /avatars/3a3182996bd41b526dcbfa8687d91963.svg
paper_authors.fullname: Kanzhi Cheng
paper_authors.type: user
paper_authors.user: cckevinn
paper_authors._id: 67d8e115f55b855ae6d8f29c
paper_authors.name: Wenpo Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:46:11.039Z
paper_authors._id: 653f8cc9d4d0924ad2404d86
paper_authors.avatarUrl: /avatars/83defb31d669393447ee04fe9989b96b.svg
paper_authors.fullname: Wenpo Song
paper_authors.type: user
paper_authors.user: songwp
paper_authors._id: 67d8e115f55b855ae6d8f29d
paper_authors.name: Jiaxin Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:24:11.342Z
paper_authors._id: 65d1d15371ca8e594f494c4f
paper_authors.avatarUrl: /avatars/80b324be013f1717f625fd7f690fed35.svg
paper_authors.fullname: Jiaxin Fan
paper_authors.type: user
paper_authors.user: DerekFan
paper_authors._id: 67d8e115f55b855ae6d8f29e
paper_authors.name: Zheng Ma
paper_authors._id: 67d8e115f55b855ae6d8f29f
paper_authors.name: Qiushi Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:25:26.087Z
paper_authors._id: 6064a0eeb1703ddba0d458b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png
paper_authors.fullname: Qiushi
paper_authors.type: user
paper_authors.user: QiushiSun
paper_authors._id: 67d8e115f55b855ae6d8f2a0
paper_authors.name: Fangzhi Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:25:14.904Z
paper_authors._id: 64e6cf78ecce34cb442dc889
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e6cf78ecce34cb442dc889/qVZFiUEpBpSkmH8SQeinm.jpeg
paper_authors.fullname: Fangzhi Xu
paper_authors.type: user
paper_authors.user: xufangzhi
paper_authors._id: 67d8e115f55b855ae6d8f2a1
paper_authors.name: Chenyang Yan
paper_authors._id: 67d8e115f55b855ae6d8f2a2
paper_authors.name: Nuo Chen
paper_authors._id: 67d8e115f55b855ae6d8f2a3
paper_authors.name: Jianbing Zhang
paper_authors._id: 67d8e115f55b855ae6d8f2a4
paper_authors.name: Jiajun Chen
paper_title: CapArena: Benchmarking and Analyzing Detailed Image Captioning in the
    LLM Era
paper_summary: Image captioning has been a longstanding challenge in vision-language
  research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate
  detailed and comprehensive image descriptions. However, benchmarking the
  quality of such captions remains unresolved. This paper addresses two key
  questions: (1) How well do current VLMs actually perform on image captioning,
  particularly compared to humans? We built CapArena, a platform with over 6000
  pairwise caption battles and high-quality human preference votes. Our
  arena-style evaluation marks a milestone, showing that leading models like
  GPT-4o achieve or even surpass human performance, while most open-source models
  lag behind. (2) Can automated metrics reliably assess detailed caption quality?
  Using human annotations from CapArena, we evaluate traditional and recent
  captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while
  some metrics (e.g., METEOR) show decent caption-level agreement with humans,
  their systematic biases lead to inconsistencies in model ranking. In contrast,
  VLM-as-a-Judge demonstrates robust discernment at both the caption and model
  levels. Building on these insights, we release CapArena-Auto, an accurate and
  efficient automated benchmark for detailed captioning, achieving 94.3%
  correlation with human rankings at just $4 per test. Data and resources will be
  open-sourced at https://caparena.github.io.
paper_ai_summary: CapArena evaluates current VLMs on image captioning, finding that models like GPT-4o surpass human performance, while automated metrics show biases; CapArena-Auto provides an accurate and efficient automated benchmark.
paper_ai_keywords: Vision-Language Models image captioning CapArena pairwise caption battles human preference votes GPT-4o METEOR VLM-as-a-Judge CapArena-Auto