CoMemo: LVLMs Need Image Context with Image Memory
Recent advancements in Large Vision-Language Models built upon Large Language
Models have established aligning visual features with LLM representations as
the dominant paradigm. However, inherited LLM architectural designs introduce
suboptimal characteristics for multimodal processing. First, LVLMs exhibit a
bimodal distribution in attention allocation, leading to the progressive
neglect of middle visual content as context expands. Second, conventional
positional encoding schemes fail to preserve vital 2D structural relationships
when processing dynamic high-resolution images. To address these limitations,
we propose CoMemo - a dual-path architecture that combines a Context image path
with an image Memory path for visual processing, effectively alleviating visual
information neglect. Additionally, we introduce RoPE-DHR, a novel positional
encoding mechanism that employs thumbnail-based positional aggregation to
maintain 2D spatial awareness while mitigating remote decay in extended
sequences. Evaluations across seven benchmarks,including long-context
comprehension, multi-image reasoning, and visual question answering,
demonstrate CoMemo's superior performance compared to conventional LVLM
architectures. Project page is available at
https://lalbj.github.io/projects/CoMemo/.
6850e0285e07650ecce890f3
Shi Liu
claimed_verified
2025-06-18T12:16:48.071Z
637f22d27119bd030dfd4af8
/avatars/55c468c40ad3bd3218d086759fb1be3c.svg
Shi Liu
user
CLLBJ16
6850e0285e07650ecce890f4
Weijie Su
claimed_verified
2025-06-19T09:10:33.557Z
63e4562f9db5da2dc1f3b520
/avatars/f4eecf1396b05e1c72436e7026d85cef.svg
Weijie Su
user
jackroos
6850e0285e07650ecce890f5
Xizhou Zhu
6850e0285e07650ecce890f6
Wenhai Wang
6850e0285e07650ecce890f7
Jifeng Dai
CoMemo: LVLMs Need Image Context with Image Memory
Recent advancements in Large Vision-Language Models built upon Large Language
Models have established aligning visual features with LLM representations as
the dominant paradigm. However, inherited LLM architectural designs introduce
suboptimal characteristics for multimodal processing. First, LVLMs exhibit a
bimodal distribution in attention allocation, leading to the progressive
neglect of middle visual content as context expands. Second, conventional
positional encoding schemes fail to preserve vital 2D structural relationships
when processing dynamic high-resolution images. To address these limitations,
we propose CoMemo - a dual-path architecture that combines a Context image path
with an image Memory path for visual processing, effectively alleviating visual
information neglect. Additionally, we introduce RoPE-DHR, a novel positional
encoding mechanism that employs thumbnail-based positional aggregation to
maintain 2D spatial awareness while mitigating remote decay in extended
sequences. Evaluations across seven benchmarks,including long-context
comprehension, multi-image reasoning, and visual question answering,
demonstrate CoMemo's superior performance compared to conventional LVLM
architectures. Project page is available at
https://lalbj.github.io/projects/CoMemo/.
CoMemo addresses visual information neglect and spatial awareness in multimodal processing by using a dual-path architecture and a novel positional encoding mechanism.
Large Vision-Language Models Large Language Models multimodal processing bimodal distribution attention allocation middle visual content positional encoding visual processing image Memory path RoPE-DHR positional aggregation 2D structural relationships spatial awareness remote decay long-context comprehension multi-image reasoning visual question answering