title: MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation
summary: Multimodal Large Language Models (MLLMs) frequently exhibit hallucination
  phenomena, but the underlying reasons remain poorly understood. In this paper,
  we present an empirical analysis and find that, although MLLMs incorrectly
  generate the objects in the final output, they are actually able to recognize
  visual objects in the preceding layers. We speculate that this may be due to
  the strong knowledge priors of the language model suppressing the visual
  information, leading to hallucinations. Motivated by this, we propose a novel
  dynamic correction decoding method for MLLMs (DeCo), which adaptively selects
  the appropriate preceding layers and proportionally integrates knowledge into
  the final layer to adjust the output logits. Note that DeCo is model agnostic
  and can be seamlessly incorporated with various classic decoding strategies and
  applied to different MLLMs. We evaluate DeCo on widely-used benchmarks,
  demonstrating that it can reduce hallucination rates by a large margin compared
  to baselines, highlighting its potential to mitigate hallucinations. Code is
  available at https://github.com/zjunlp/DeCo.
paper_authors._id: 670f31c4e218fd7c74db69ad
paper_authors.name: Chenxi Wang
paper_authors._id: 670f31c4e218fd7c74db69ae
paper_authors.name: Xiang Chen
paper_authors._id: 670f31c4e218fd7c74db69af
paper_authors.name: Ningyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:43:40.404Z
paper_authors._id: 620b3bbb0668e435407c8d0a
paper_authors.avatarUrl: /avatars/e0fccbb2577d76088e09f054c35cffbc.svg
paper_authors.fullname: Ningyu Zhang
paper_authors.type: user
paper_authors.user: Ningyu
paper_authors._id: 670f31c4e218fd7c74db69b0
paper_authors.name: Bozhong Tian
paper_authors._id: 670f31c4e218fd7c74db69b1
paper_authors.name: Haoming Xu
paper_authors._id: 670f31c4e218fd7c74db69b2
paper_authors.name: Shumin Deng
paper_authors._id: 670f31c4e218fd7c74db69b3
paper_authors.name: Huajun Chen
paper_title: MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation
paper_summary: Multimodal Large Language Models (MLLMs) frequently exhibit hallucination
  phenomena, but the underlying reasons remain poorly understood. In this paper,
  we present an empirical analysis and find that, although MLLMs incorrectly
  generate the objects in the final output, they are actually able to recognize
  visual objects in the preceding layers. We speculate that this may be due to
  the strong knowledge priors of the language model suppressing the visual
  information, leading to hallucinations. Motivated by this, we propose a novel
  dynamic correction decoding method for MLLMs (DeCo), which adaptively selects
  the appropriate preceding layers and proportionally integrates knowledge into
  the final layer to adjust the output logits. Note that DeCo is model agnostic
  and can be seamlessly incorporated with various classic decoding strategies and
  applied to different MLLMs. We evaluate DeCo on widely-used benchmarks,
  demonstrating that it can reduce hallucination rates by a large margin compared
  to baselines, highlighting its potential to mitigate hallucinations. Code is
  available at https://github.com/zjunlp/DeCo.
paper_ai_summary: A novel dynamic correction decoding method for multimodal large language models reduces hallucination rates by integrating knowledge from preceding layers into the final output.
paper_ai_keywords: multimodal large language models hallucination phenomena dynamic correction decoding preceding layers knowledge priors output logits model agnostic classic decoding strategies