title: Seeing is Believing, but How Much? A Comprehensive Analysis of
    Verbalized Calibration in Vision-Language Models
summary: Uncertainty quantification is essential for assessing the reliability and
  trustworthiness of modern AI systems. Among existing approaches, verbalized
  uncertainty, where models express their confidence through natural language,
  has emerged as a lightweight and interpretable solution in large language
  models (LLMs). However, its effectiveness in vision-language models (VLMs)
  remains insufficiently studied. In this work, we conduct a comprehensive
  evaluation of verbalized confidence in VLMs, spanning three model categories,
  four task domains, and three evaluation scenarios. Our results show that
  current VLMs often display notable miscalibration across diverse tasks and
  settings. Notably, visual reasoning models (i.e., thinking with images)
  consistently exhibit better calibration, suggesting that modality-specific
  reasoning is critical for reliable uncertainty estimation. To further address
  calibration challenges, we introduce Visual Confidence-Aware Prompting, a
  two-stage prompting strategy that improves confidence alignment in multimodal
  settings. Overall, our study highlights the inherent miscalibration in VLMs
  across modalities. More broadly, our findings underscore the fundamental
  importance of modality alignment and model faithfulness in advancing reliable
  multimodal systems.
paper_authors._id: 68365bc843cbbe13e0fabdad
paper_authors.name: Weihao Xuan
paper_authors._id: 68365bc843cbbe13e0fabdae
paper_authors.name: Qingcheng Zeng
paper_authors._id: 68365bc843cbbe13e0fabdaf
paper_authors.name: Heli Qi
paper_authors._id: 68365bc843cbbe13e0fabdb0
paper_authors.name: Junjue Wang
paper_authors._id: 68365bc843cbbe13e0fabdb1
paper_authors.name: Naoto Yokoya
paper_title: Seeing is Believing, but How Much? A Comprehensive Analysis of
    Verbalized Calibration in Vision-Language Models
paper_summary: Uncertainty quantification is essential for assessing the reliability and
  trustworthiness of modern AI systems. Among existing approaches, verbalized
  uncertainty, where models express their confidence through natural language,
  has emerged as a lightweight and interpretable solution in large language
  models (LLMs). However, its effectiveness in vision-language models (VLMs)
  remains insufficiently studied. In this work, we conduct a comprehensive
  evaluation of verbalized confidence in VLMs, spanning three model categories,
  four task domains, and three evaluation scenarios. Our results show that
  current VLMs often display notable miscalibration across diverse tasks and
  settings. Notably, visual reasoning models (i.e., thinking with images)
  consistently exhibit better calibration, suggesting that modality-specific
  reasoning is critical for reliable uncertainty estimation. To further address
  calibration challenges, we introduce Visual Confidence-Aware Prompting, a
  two-stage prompting strategy that improves confidence alignment in multimodal
  settings. Overall, our study highlights the inherent miscalibration in VLMs
  across modalities. More broadly, our findings underscore the fundamental
  importance of modality alignment and model faithfulness in advancing reliable
  multimodal systems.
paper_ai_summary: The study evaluates verbalized confidence in vision-language models, revealing miscalibration across tasks and suggesting that modality-specific reasoning and Visual Confidence-Aware Prompting can improve reliability.
paper_ai_keywords: vision-language models verbalized uncertainty natural language visual reasoning models multimodal settings confidence calibration Visual Confidence-Aware Prompting modality alignment model faithfulness