title: Painting with Words: Elevating Detailed Image Captioning with Benchmark
    and Alignment Learning
summary: Image captioning has long been a pivotal task in visual understanding, with
  recent advancements in vision-language models (VLMs) significantly enhancing
  the ability to generate detailed image captions. However, the evaluation of
  detailed image captioning remains underexplored due to outdated evaluation
  metrics and coarse annotations. In this paper, we introduce DeCapBench along
  with a novel metric, DCScore, specifically designed for detailed captioning
  tasks. DCScore evaluates hallucinations and fine-grained comprehensiveness by
  deconstructing responses into the smallest self-sufficient units, termed
  primitive information units, and assessing them individually. Our evaluation
  shows that DCScore aligns more closely with human judgment than other
  rule-based or model-based metrics. Concurrently, DeCapBench exhibits a high
  correlation with VLM arena results on descriptive tasks, surpassing existing
  benchmarks for vision-language models. Additionally, we present an automatic
  fine-grained feedback collection method, FeedQuill, for preference optimization
  based on our advanced metric, showing robust generalization capabilities across
  auto-generated preference data. Extensive experiments on multiple VLMs
  demonstrate that our method not only significantly reduces hallucinations but
  also enhances performance across various benchmarks, achieving superior detail
  captioning performance while surpassing GPT-4o.
paper_authors._id: 67dde94f169ce083a5098930
paper_authors.name: Qinghao Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:18:44.811Z
paper_authors._id: 64530fc01a57e1179c1fe4c0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0lncTpIHXn6suB0p-oSma.jpeg
paper_authors.fullname: QinghaoYe
paper_authors.type: user
paper_authors.user: MAGAer13
paper_authors._id: 67dde94f169ce083a5098931
paper_authors.name: Xianhan Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-13T08:13:29.058Z
paper_authors._id: 666d10506ec1340a7421a92d
paper_authors.avatarUrl: /avatars/5bff16fca817b35760e0c26d8d7c9424.svg
paper_authors.fullname: Xianhan Zeng
paper_authors.type: user
paper_authors.user: RitzzZz23
paper_authors._id: 67dde94f169ce083a5098932
paper_authors.name: Fu Li
paper_authors._id: 67dde94f169ce083a5098933
paper_authors.name: Chunyuan Li
paper_authors._id: 67dde94f169ce083a5098934
paper_authors.name: Haoqi Fan
paper_title: Painting with Words: Elevating Detailed Image Captioning with Benchmark
    and Alignment Learning
paper_summary: Image captioning has long been a pivotal task in visual understanding, with
  recent advancements in vision-language models (VLMs) significantly enhancing
  the ability to generate detailed image captions. However, the evaluation of
  detailed image captioning remains underexplored due to outdated evaluation
  metrics and coarse annotations. In this paper, we introduce DeCapBench along
  with a novel metric, DCScore, specifically designed for detailed captioning
  tasks. DCScore evaluates hallucinations and fine-grained comprehensiveness by
  deconstructing responses into the smallest self-sufficient units, termed
  primitive information units, and assessing them individually. Our evaluation
  shows that DCScore aligns more closely with human judgment than other
  rule-based or model-based metrics. Concurrently, DeCapBench exhibits a high
  correlation with VLM arena results on descriptive tasks, surpassing existing
  benchmarks for vision-language models. Additionally, we present an automatic
  fine-grained feedback collection method, FeedQuill, for preference optimization
  based on our advanced metric, showing robust generalization capabilities across
  auto-generated preference data. Extensive experiments on multiple VLMs
  demonstrate that our method not only significantly reduces hallucinations but
  also enhances performance across various benchmarks, achieving superior detail
  captioning performance while surpassing GPT-4o.
paper_ai_summary: DeCapBench, a new benchmark with DCScore, improves detailed image captioning evaluation and performance by reducing hallucinations and enhancing comprehensiveness.
paper_ai_keywords: vision-language models DeCapBench DCScore primitive information units hallucinations fine-grained comprehensiveness FeedQuill preference optimization auto-generated preference data