title: DASH: Detection and Assessment of Systematic Hallucinations of VLMs
summary: Vision-language models (VLMs) are prone to object hallucinations, where they
  erroneously indicate the presenceof certain objects in an image. Existing
  benchmarks quantify hallucinations using relatively small, labeled datasets.
  However, this approach is i) insufficient to assess hallucinations that arise
  in open-world settings, where VLMs are widely used, and ii) inadequate for
  detecting systematic errors in VLMs. We propose DASH (Detection and Assessment
  of Systematic Hallucinations), an automatic, large-scale pipeline designed to
  identify systematic hallucinations of VLMs on real-world images in an
  open-world setting. A key component is DASH-OPT for image-based retrieval,
  where we optimize over the ''natural image manifold'' to generate images that
  mislead the VLM. The output of DASH consists of clusters of real and
  semantically similar images for which the VLM hallucinates an object. We apply
  DASH to PaliGemma and two LLaVA-NeXT models across 380 object classes and, in
  total, find more than 19k clusters with 950k images. We study the transfer of
  the identified systematic hallucinations to other VLMs and show that
  fine-tuning PaliGemma with the model-specific images obtained with DASH
  mitigates object hallucinations. Code and data are available at
  https://YanNeu.github.io/DASH.
paper_authors._id: 67ecf726f87008271b1d07d5
paper_authors.name: Maximilian Augustin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T14:05:03.470Z
paper_authors._id: 67d19d9a3e04b5840ab8684f
paper_authors.avatarUrl: /avatars/58b2e28c7360b546141a3cd91a392eed.svg
paper_authors.fullname: Maximilian Augustin
paper_authors.type: user
paper_authors.user: M4ximal
paper_authors._id: 67ecf726f87008271b1d07d6
paper_authors.name: Yannic Neuhaus
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:51:14.837Z
paper_authors._id: 650971dbce83a0c12a851000
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ec7H2cZpaZz07W02FT5U9.png
paper_authors.fullname: Yannic Neuhaus
paper_authors.type: user
paper_authors.user: YanNeu
paper_authors._id: 67ecf726f87008271b1d07d7
paper_authors.name: Matthias Hein
paper_title: DASH: Detection and Assessment of Systematic Hallucinations of VLMs
paper_summary: Vision-language models (VLMs) are prone to object hallucinations, where they
  erroneously indicate the presenceof certain objects in an image. Existing
  benchmarks quantify hallucinations using relatively small, labeled datasets.
  However, this approach is i) insufficient to assess hallucinations that arise
  in open-world settings, where VLMs are widely used, and ii) inadequate for
  detecting systematic errors in VLMs. We propose DASH (Detection and Assessment
  of Systematic Hallucinations), an automatic, large-scale pipeline designed to
  identify systematic hallucinations of VLMs on real-world images in an
  open-world setting. A key component is DASH-OPT for image-based retrieval,
  where we optimize over the ''natural image manifold'' to generate images that
  mislead the VLM. The output of DASH consists of clusters of real and
  semantically similar images for which the VLM hallucinates an object. We apply
  DASH to PaliGemma and two LLaVA-NeXT models across 380 object classes and, in
  total, find more than 19k clusters with 950k images. We study the transfer of
  the identified systematic hallucinations to other VLMs and show that
  fine-tuning PaliGemma with the model-specific images obtained with DASH
  mitigates object hallucinations. Code and data are available at
  https://YanNeu.github.io/DASH.
paper_ai_summary: DASH, an automatic pipeline, identifies and assesses systematic hallucinations in vision-language models across a large dataset, improving model accuracy through fine-tuning with misleading images generated from the natural image manifold.
paper_ai_keywords: Detection and Assessment of Systematic Hallucinations DASH DASH-OPT natural image manifold vision-language models object hallucinations fine-tuning