title: The Hidden Life of Tokens: Reducing Hallucination of Large
    Vision-Language Models via Visual Information Steering
summary: Large Vision-Language Models (LVLMs) can reason effectively over both textual
  and visual inputs, but they tend to hallucinate syntactically coherent yet
  visually ungrounded contents. In this paper, we investigate the internal
  dynamics of hallucination by examining the tokens logits rankings throughout
  the generation process, revealing three key patterns in how LVLMs process
  information: (1) gradual visual information loss -- visually grounded tokens
  gradually become less favored throughout generation, and (2) early excitation
  -- semantically meaningful tokens achieve peak activation in the layers earlier
  than the final layer. (3) hidden genuine information -- visually grounded
  tokens though not being eventually decided still retain relatively high
  rankings at inference. Based on these insights, we propose VISTA (Visual
  Information Steering with Token-logit Augmentation), a training-free
  inference-time intervention framework that reduces hallucination while
  promoting genuine information. VISTA works by combining two complementary
  approaches: reinforcing visual information in activation space and leveraging
  early layer activations to promote semantically meaningful decoding. Compared
  to existing methods, VISTA requires no external supervision and is applicable
  to various decoding strategies. Extensive experiments show that VISTA on
  average reduces hallucination by abount 40% on evaluated open-ended generation
  task, and it consistently outperforms existing methods on four benchmarks
  across four architectures under three decoding strategies.
paper_authors._id: 67aab82e6024056209d727a8
paper_authors.name: Zhuowei Li
paper_authors._id: 67aab82e6024056209d727a9
paper_authors.name: Haizhou Shi
paper_authors._id: 67aab82e6024056209d727aa
paper_authors.name: Yunhe Gao
paper_authors._id: 67aab82e6024056209d727ab
paper_authors.name: Di Liu
paper_authors._id: 67aab82e6024056209d727ac
paper_authors.name: Zhenting Wang
paper_authors._id: 67aab82e6024056209d727ad
paper_authors.name: Yuxiao Chen
paper_authors._id: 67aab82e6024056209d727ae
paper_authors.name: Ting Liu
paper_authors._id: 67aab82e6024056209d727af
paper_authors.name: Long Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:34:47.676Z
paper_authors._id: 650c249887dcda6616baa040
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tIVfxoAHAJ0sWyDMkaarA.jpeg
paper_authors.fullname: Long Zhao
paper_authors.type: user
paper_authors.user: garyzhao9012
paper_authors._id: 67aab82e6024056209d727b0
paper_authors.name: Hao Wang
paper_authors._id: 67aab82e6024056209d727b1
paper_authors.name: Dimitris N. Metaxas
paper_title: The Hidden Life of Tokens: Reducing Hallucination of Large
    Vision-Language Models via Visual Information Steering
paper_summary: Large Vision-Language Models (LVLMs) can reason effectively over both textual
  and visual inputs, but they tend to hallucinate syntactically coherent yet
  visually ungrounded contents. In this paper, we investigate the internal
  dynamics of hallucination by examining the tokens logits rankings throughout
  the generation process, revealing three key patterns in how LVLMs process
  information: (1) gradual visual information loss -- visually grounded tokens
  gradually become less favored throughout generation, and (2) early excitation
  -- semantically meaningful tokens achieve peak activation in the layers earlier
  than the final layer. (3) hidden genuine information -- visually grounded
  tokens though not being eventually decided still retain relatively high
  rankings at inference. Based on these insights, we propose VISTA (Visual
  Information Steering with Token-logit Augmentation), a training-free
  inference-time intervention framework that reduces hallucination while
  promoting genuine information. VISTA works by combining two complementary
  approaches: reinforcing visual information in activation space and leveraging
  early layer activations to promote semantically meaningful decoding. Compared
  to existing methods, VISTA requires no external supervision and is applicable
  to various decoding strategies. Extensive experiments show that VISTA on
  average reduces hallucination by abount 40% on evaluated open-ended generation
  task, and it consistently outperforms existing methods on four benchmarks
  across four architectures under three decoding strategies.
paper_ai_summary: VISTA reduces hallucination in large vision-language models by steering visual information during inference without additional training.
paper_ai_keywords: LVLMs hallucination tokens logits rankings visual information loss early excitation hidden genuine information VISTA activation space semantically meaningful decoding decoding strategies open-ended generation tasks benchmarks architectures