Mitigating Object Hallucination via Concentric Causal Attention
Recent Large Vision Language Models (LVLMs) present remarkable zero-shot
conversational and reasoning capabilities given multimodal queries.
Nevertheless, they suffer from object hallucination, a phenomenon where LVLMs
are prone to generate textual responses not factually aligned with image
inputs. Our pilot study reveals that object hallucination is closely tied with
Rotary Position Encoding (RoPE), a widely adopted positional dependency
modeling design in existing LVLMs. Due to the long-term decay in RoPE, LVLMs
tend to hallucinate more when relevant visual cues are distant from instruction
tokens in the multimodal input sequence. Additionally, we observe a similar
effect when reversing the sequential order of visual tokens during multimodal
alignment. Our tests indicate that long-term decay in RoPE poses challenges to
LVLMs while capturing visual-instruction interactions across long distances. We
propose Concentric Causal Attention (CCA), a simple yet effective positional
alignment strategy that mitigates the impact of RoPE long-term decay in LVLMs
by naturally reducing relative distance between visual and instruction tokens.
With CCA, visual tokens can better interact with instruction tokens, thereby
enhancing model's perception capability and alleviating object hallucination.
Without bells and whistles, our positional alignment method surpasses existing
hallucination mitigation strategies by large margins on multiple object
hallucination benchmarks.
67171f82e90685660e1b2fc8
Yun Xing
claimed_verified
2024-10-22T07:57:04.026Z
65386066bf790526ef358701
/avatars/f02b48f1b05c353e5ab371de45b9cf4c.svg
Xing Yun
user
xing0047
67171f82e90685660e1b2fc9
Yiheng Li
67171f82e90685660e1b2fca
Ivan Laptev
67171f82e90685660e1b2fcb
Shijian Lu
Mitigating Object Hallucination via Concentric Causal Attention
Recent Large Vision Language Models (LVLMs) present remarkable zero-shot
conversational and reasoning capabilities given multimodal queries.
Nevertheless, they suffer from object hallucination, a phenomenon where LVLMs
are prone to generate textual responses not factually aligned with image
inputs. Our pilot study reveals that object hallucination is closely tied with
Rotary Position Encoding (RoPE), a widely adopted positional dependency
modeling design in existing LVLMs. Due to the long-term decay in RoPE, LVLMs
tend to hallucinate more when relevant visual cues are distant from instruction
tokens in the multimodal input sequence. Additionally, we observe a similar
effect when reversing the sequential order of visual tokens during multimodal
alignment. Our tests indicate that long-term decay in RoPE poses challenges to
LVLMs while capturing visual-instruction interactions across long distances. We
propose Concentric Causal Attention (CCA), a simple yet effective positional
alignment strategy that mitigates the impact of RoPE long-term decay in LVLMs
by naturally reducing relative distance between visual and instruction tokens.
With CCA, visual tokens can better interact with instruction tokens, thereby
enhancing model's perception capability and alleviating object hallucination.
Without bells and whistles, our positional alignment method surpasses existing
hallucination mitigation strategies by large margins on multiple object
hallucination benchmarks.
Concentric Causal Attention (CCA) mitigates object hallucination in Large Vision Language Models by addressing long-term decay in Rotary Position Encoding (RoPE) during multimodal interactions.
Large Vision Language Models (LVLMs) object hallucination Rotary Position Encoding (RoPE) long-term decay multimodal alignment Concentric Causal Attention (CCA) positional alignment visual-instruction interactions perception capability