title: Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and
    Dual Evaluation Metrics for Factuality and Coverage
summary: Multimodal large language models (MLLMs) excel at generating highly detailed
  captions but often produce hallucinations. Our analysis reveals that existing
  hallucination detection methods struggle with detailed captions. We attribute
  this to the increasing reliance of MLLMs on their generated text, rather than
  the input image, as the sequence length grows. To address this issue, we
  propose a multiagent approach that leverages LLM-MLLM collaboration to correct
  given captions. Additionally, we introduce an evaluation framework and a
  benchmark dataset to facilitate the systematic analysis of detailed captions.
  Our experiments demonstrate that our proposed evaluation method better aligns
  with human judgments of factuality than existing metrics and that existing
  approaches to improve the MLLM factuality may fall short in hyper-detailed
  image captioning tasks. In contrast, our proposed method significantly enhances
  the factual accuracy of captions, even improving those generated by GPT-4V.
  Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating
  that an MLLM's performance on VQA benchmarks may not correlate with its ability
  to generate detailed image captions.
paper_authors._id: 6768c08c75d8e8d042beda80
paper_authors.name: Saehyung Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:10:04.150Z
paper_authors._id: 6768bf111d652acb0c60b938
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/5j0JkrslB8zGf4Wy8tKFc.png
paper_authors.fullname: Saehyung Lee
paper_authors.type: user
paper_authors.user: saehyungl
paper_authors._id: 6768c08c75d8e8d042beda81
paper_authors.name: Seunghyun Yoon
paper_authors._id: 6768c08c75d8e8d042beda82
paper_authors.name: Trung Bui
paper_authors._id: 6768c08c75d8e8d042beda83
paper_authors.name: Jing Shi
paper_authors._id: 6768c08c75d8e8d042beda84
paper_authors.name: Sungroh Yoon
paper_title: Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and
    Dual Evaluation Metrics for Factuality and Coverage
paper_summary: Multimodal large language models (MLLMs) excel at generating highly detailed
  captions but often produce hallucinations. Our analysis reveals that existing
  hallucination detection methods struggle with detailed captions. We attribute
  this to the increasing reliance of MLLMs on their generated text, rather than
  the input image, as the sequence length grows. To address this issue, we
  propose a multiagent approach that leverages LLM-MLLM collaboration to correct
  given captions. Additionally, we introduce an evaluation framework and a
  benchmark dataset to facilitate the systematic analysis of detailed captions.
  Our experiments demonstrate that our proposed evaluation method better aligns
  with human judgments of factuality than existing metrics and that existing
  approaches to improve the MLLM factuality may fall short in hyper-detailed
  image captioning tasks. In contrast, our proposed method significantly enhances
  the factual accuracy of captions, even improving those generated by GPT-4V.
  Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating
  that an MLLM's performance on VQA benchmarks may not correlate with its ability
  to generate detailed image captions.
paper_ai_summary: A multiagent approach using LLM-MLLM collaboration improves the factual accuracy of detailed image captions, surpassing existing methods and highlighting the limitations of VQA benchmarks.
paper_ai_keywords: multimodal large language models MLLMs hallucination detection sequence length multiagent approach LLM-MLLM collaboration evaluation framework benchmark dataset factuality GPT-4V VQA benchmarks image captioning