CapArena: Benchmarking and Analyzing Detailed Image Captioning in the
  LLM Era
Image captioning has been a longstanding challenge in vision-language
research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate
detailed and comprehensive image descriptions. However, benchmarking the
quality of such captions remains unresolved. This paper addresses two key
questions: (1) How well do current VLMs actually perform on image captioning,
particularly compared to humans? We built CapArena, a platform with over 6000
pairwise caption battles and high-quality human preference votes. Our
arena-style evaluation marks a milestone, showing that leading models like
GPT-4o achieve or even surpass human performance, while most open-source models
lag behind. (2) Can automated metrics reliably assess detailed caption quality?
Using human annotations from CapArena, we evaluate traditional and recent
captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while
some metrics (e.g., METEOR) show decent caption-level agreement with humans,
their systematic biases lead to inconsistencies in model ranking. In contrast,
VLM-as-a-Judge demonstrates robust discernment at both the caption and model
levels. Building on these insights, we release CapArena-Auto, an accurate and
efficient automated benchmark for detailed captioning, achieving 94.3%
correlation with human rankings at just $4 per test. Data and resources will be
open-sourced at https://caparena.github.io.
67d8e115f55b855ae6d8f29b
Kanzhi Cheng
claimed_verified
2025-03-18T11:34:32.729Z
63340dbbd92c5842ae71d1e9
/avatars/3a3182996bd41b526dcbfa8687d91963.svg
Kanzhi Cheng
user
cckevinn
67d8e115f55b855ae6d8f29c
Wenpo Song
claimed_verified
2025-03-19T09:46:11.039Z
653f8cc9d4d0924ad2404d86
/avatars/83defb31d669393447ee04fe9989b96b.svg
Wenpo Song
user
songwp
67d8e115f55b855ae6d8f29d
Jiaxin Fan
admin_assigned
2025-03-19T13:24:11.342Z
65d1d15371ca8e594f494c4f
/avatars/80b324be013f1717f625fd7f690fed35.svg
Jiaxin Fan
user
DerekFan
67d8e115f55b855ae6d8f29e
Zheng Ma
67d8e115f55b855ae6d8f29f
Qiushi Sun
admin_assigned
2025-03-19T13:25:26.087Z
6064a0eeb1703ddba0d458b9
https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png
Qiushi
user
QiushiSun
67d8e115f55b855ae6d8f2a0
Fangzhi Xu
admin_assigned
2025-03-19T13:25:14.904Z
64e6cf78ecce34cb442dc889
https://cdn-avatars.huggingface.co/v1/production/uploads/64e6cf78ecce34cb442dc889/qVZFiUEpBpSkmH8SQeinm.jpeg
Fangzhi Xu
user
xufangzhi
67d8e115f55b855ae6d8f2a1
Chenyang Yan
67d8e115f55b855ae6d8f2a2
Nuo Chen
67d8e115f55b855ae6d8f2a3
Jianbing Zhang
67d8e115f55b855ae6d8f2a4
Jiajun Chen
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the
  LLM Era
Image captioning has been a longstanding challenge in vision-language
research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate
detailed and comprehensive image descriptions. However, benchmarking the
quality of such captions remains unresolved. This paper addresses two key
questions: (1) How well do current VLMs actually perform on image captioning,
particularly compared to humans? We built CapArena, a platform with over 6000
pairwise caption battles and high-quality human preference votes. Our
arena-style evaluation marks a milestone, showing that leading models like
GPT-4o achieve or even surpass human performance, while most open-source models
lag behind. (2) Can automated metrics reliably assess detailed caption quality?
Using human annotations from CapArena, we evaluate traditional and recent
captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while
some metrics (e.g., METEOR) show decent caption-level agreement with humans,
their systematic biases lead to inconsistencies in model ranking. In contrast,
VLM-as-a-Judge demonstrates robust discernment at both the caption and model
levels. Building on these insights, we release CapArena-Auto, an accurate and
efficient automated benchmark for detailed captioning, achieving 94.3%
correlation with human rankings at just $4 per test. Data and resources will be
open-sourced at https://caparena.github.io.
CapArena evaluates current VLMs on image captioning, finding that models like GPT-4o surpass human performance, while automated metrics show biases; CapArena-Auto provides an accurate and efficient automated benchmark.
Vision-Language Models image captioning CapArena pairwise caption battles human preference votes GPT-4o METEOR VLM-as-a-Judge CapArena-Auto