title: Woodpecker: Hallucination Correction for Multimodal Large Language
    Models
summary: Hallucination is a big shadow hanging over the rapidly evolving Multimodal
  Large Language Models (MLLMs), referring to the phenomenon that the generated
  text is inconsistent with the image content. In order to mitigate
  hallucinations, existing studies mainly resort to an instruction-tuning manner
  that requires retraining the models with specific data. In this paper, we pave
  a different way, introducing a training-free method named Woodpecker. Like a
  woodpecker heals trees, it picks out and corrects hallucinations from the
  generated text. Concretely, Woodpecker consists of five stages: key concept
  extraction, question formulation, visual knowledge validation, visual claim
  generation, and hallucination correction. Implemented in a post-remedy manner,
  Woodpecker can easily serve different MLLMs, while being interpretable by
  accessing intermediate outputs of the five stages. We evaluate Woodpecker both
  quantitatively and qualitatively and show the huge potential of this new
  paradigm. On the POPE benchmark, our method obtains a 30.66%/24.33% improvement
  in accuracy over the baseline MiniGPT-4/mPLUG-Owl. The source code is released
  at https://github.com/BradyFU/Woodpecker.
paper_authors._id: 653870d0db57c222437d9363
paper_authors.name: Shukang Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:40:27.831Z
paper_authors._id: 62d12046683d9b0360ffdf1e
paper_authors.avatarUrl: /avatars/3b79b73d897cbb5176887fe579d392d7.svg
paper_authors.fullname: Shukang Yin
paper_authors.type: user
paper_authors.user: xjtupanda
paper_authors._id: 653870d0db57c222437d9364
paper_authors.name: Chaoyou Fu
paper_authors._id: 653870d0db57c222437d9365
paper_authors.name: Sirui Zhao
paper_authors._id: 653870d0db57c222437d9366
paper_authors.name: Tong Xu
paper_authors._id: 653870d0db57c222437d9367
paper_authors.name: Hao Wang
paper_authors._id: 653870d0db57c222437d9368
paper_authors.name: Dianbo Sui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:41:33.494Z
paper_authors._id: 643e8b572263cdc630f80fee
paper_authors.avatarUrl: /avatars/93b11c28a58c5db71afd44e782493140.svg
paper_authors.fullname: Dianbo Sui
paper_authors.type: user
paper_authors.user: suidianbo
paper_authors._id: 653870d0db57c222437d9369
paper_authors.name: Yunhang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:41:48.338Z
paper_authors._id: 6483143902f98c3f05aff915
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
paper_authors.fullname: 沈云航 Yunhang Shen
paper_authors.type: user
paper_authors.user: shenyunhang
paper_authors._id: 653870d0db57c222437d936a
paper_authors.name: Ke Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:13:06.036Z
paper_authors._id: 63280915eeee4dd858083092
paper_authors.avatarUrl: /avatars/78347af4af42527d53e88d9969c5c934.svg
paper_authors.fullname: Ke Li
paper_authors.type: user
paper_authors.user: tristanli
paper_authors._id: 653870d0db57c222437d936b
paper_authors.name: Xing Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-20T13:51:53.808Z
paper_authors._id: 647401e50da364bd0d002f2a
paper_authors.avatarUrl: /avatars/f1586f610f21ddb4f868856208c2cfab.svg
paper_authors.fullname: XING Sun
paper_authors.type: user
paper_authors.user: tedsun
paper_authors._id: 653870d0db57c222437d936c
paper_authors.name: Enhong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:42:39.040Z
paper_authors._id: 64dde9ccd1885643ca14b2d4
paper_authors.avatarUrl: /avatars/04d83c2c7477212ad55eec24aa69eaa2.svg
paper_authors.fullname: EnHong Chen
paper_authors.type: user
paper_authors.user: Hongni
paper_title: Woodpecker: Hallucination Correction for Multimodal Large Language
    Models
paper_summary: Hallucination is a big shadow hanging over the rapidly evolving Multimodal
  Large Language Models (MLLMs), referring to the phenomenon that the generated
  text is inconsistent with the image content. In order to mitigate
  hallucinations, existing studies mainly resort to an instruction-tuning manner
  that requires retraining the models with specific data. In this paper, we pave
  a different way, introducing a training-free method named Woodpecker. Like a
  woodpecker heals trees, it picks out and corrects hallucinations from the
  generated text. Concretely, Woodpecker consists of five stages: key concept
  extraction, question formulation, visual knowledge validation, visual claim
  generation, and hallucination correction. Implemented in a post-remedy manner,
  Woodpecker can easily serve different MLLMs, while being interpretable by
  accessing intermediate outputs of the five stages. We evaluate Woodpecker both
  quantitatively and qualitatively and show the huge potential of this new
  paradigm. On the POPE benchmark, our method obtains a 30.66%/24.33% improvement
  in accuracy over the baseline MiniGPT-4/mPLUG-Owl. The source code is released
  at https://github.com/BradyFU/Woodpecker.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) hallucinations instruction-tuning post-remedy manner key concept extraction question formulation visual knowledge validation visual claim generation hallucination correction intermediate outputs Med30.66%/24.33% improvement POPE benchmark MiniGPT-4 mPLUG-Owl