title: Mitigating Object Hallucination via Concentric Causal Attention
summary: Recent Large Vision Language Models (LVLMs) present remarkable zero-shot
  conversational and reasoning capabilities given multimodal queries.
  Nevertheless, they suffer from object hallucination, a phenomenon where LVLMs
  are prone to generate textual responses not factually aligned with image
  inputs. Our pilot study reveals that object hallucination is closely tied with
  Rotary Position Encoding (RoPE), a widely adopted positional dependency
  modeling design in existing LVLMs. Due to the long-term decay in RoPE, LVLMs
  tend to hallucinate more when relevant visual cues are distant from instruction
  tokens in the multimodal input sequence. Additionally, we observe a similar
  effect when reversing the sequential order of visual tokens during multimodal
  alignment. Our tests indicate that long-term decay in RoPE poses challenges to
  LVLMs while capturing visual-instruction interactions across long distances. We
  propose Concentric Causal Attention (CCA), a simple yet effective positional
  alignment strategy that mitigates the impact of RoPE long-term decay in LVLMs
  by naturally reducing relative distance between visual and instruction tokens.
  With CCA, visual tokens can better interact with instruction tokens, thereby
  enhancing model's perception capability and alleviating object hallucination.
  Without bells and whistles, our positional alignment method surpasses existing
  hallucination mitigation strategies by large margins on multiple object
  hallucination benchmarks.
paper_authors._id: 67171f82e90685660e1b2fc8
paper_authors.name: Yun Xing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T07:57:04.026Z
paper_authors._id: 65386066bf790526ef358701
paper_authors.avatarUrl: /avatars/f02b48f1b05c353e5ab371de45b9cf4c.svg
paper_authors.fullname: Xing Yun
paper_authors.type: user
paper_authors.user: xing0047
paper_authors._id: 67171f82e90685660e1b2fc9
paper_authors.name: Yiheng Li
paper_authors._id: 67171f82e90685660e1b2fca
paper_authors.name: Ivan Laptev
paper_authors._id: 67171f82e90685660e1b2fcb
paper_authors.name: Shijian Lu
paper_title: Mitigating Object Hallucination via Concentric Causal Attention
paper_summary: Recent Large Vision Language Models (LVLMs) present remarkable zero-shot
  conversational and reasoning capabilities given multimodal queries.
  Nevertheless, they suffer from object hallucination, a phenomenon where LVLMs
  are prone to generate textual responses not factually aligned with image
  inputs. Our pilot study reveals that object hallucination is closely tied with
  Rotary Position Encoding (RoPE), a widely adopted positional dependency
  modeling design in existing LVLMs. Due to the long-term decay in RoPE, LVLMs
  tend to hallucinate more when relevant visual cues are distant from instruction
  tokens in the multimodal input sequence. Additionally, we observe a similar
  effect when reversing the sequential order of visual tokens during multimodal
  alignment. Our tests indicate that long-term decay in RoPE poses challenges to
  LVLMs while capturing visual-instruction interactions across long distances. We
  propose Concentric Causal Attention (CCA), a simple yet effective positional
  alignment strategy that mitigates the impact of RoPE long-term decay in LVLMs
  by naturally reducing relative distance between visual and instruction tokens.
  With CCA, visual tokens can better interact with instruction tokens, thereby
  enhancing model's perception capability and alleviating object hallucination.
  Without bells and whistles, our positional alignment method surpasses existing
  hallucination mitigation strategies by large margins on multiple object
  hallucination benchmarks.
paper_ai_keywords: Large Vision Language Models (LVLMs) zero-shot conversational reasoning capabilities multimodal queries object hallucination Rotary Position Encoding (RoPE) positional dependency modeling long-term decay multimodal input sequence sequential order multimodal alignment capturing visual-instruction interactions Concentric Causal Attention (CCA) positional alignment strategy