title: HoPE: Hybrid of Position Embedding for Length Generalization in
    Vision-Language Models
summary: Vision-Language Models (VLMs) have made significant progress in multimodal
  tasks. However, their performance often deteriorates in long-context scenarios,
  particularly long videos. While Rotary Position Embedding (RoPE) has been
  widely adopted for length generalization in Large Language Models (LLMs),
  extending vanilla RoPE to capture the intricate spatial-temporal dependencies
  in videos remains an unsolved challenge. Existing methods typically allocate
  different frequencies within RoPE to encode 3D positional information. However,
  these allocation strategies mainly rely on heuristics, lacking in-depth
  theoretical analysis. In this paper, we first study how different allocation
  strategies impact the long-context capabilities of VLMs. Our analysis reveals
  that current multimodal RoPEs fail to reliably capture semantic similarities
  over extended contexts. To address this issue, we propose HoPE, a Hybrid of
  Position Embedding designed to improve the long-context capabilities of VLMs.
  HoPE introduces a hybrid frequency allocation strategy for reliable semantic
  modeling over arbitrarily long context, and a dynamic temporal scaling
  mechanism to facilitate robust learning and flexible inference across diverse
  context lengths. Extensive experiments across four video benchmarks on long
  video understanding and retrieval tasks demonstrate that HoPE consistently
  outperforms existing methods, confirming its effectiveness. Code is available
  at https://github.com/hrlics/HoPE.
paper_authors._id: 68385abc8c8e3b721702d006
paper_authors.name: Haoran Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T20:19:23.117Z
paper_authors._id: 64e86d6e7ef2c72cbcdb27b7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e86d6e7ef2c72cbcdb27b7/RLuJJ1xuO665LMWneVDZz.jpeg
paper_authors.fullname: Haoran Li
paper_authors.type: user
paper_authors.user: brian13
paper_authors._id: 68385abc8c8e3b721702d007
paper_authors.name: Yingjie Qin
paper_authors._id: 68385abc8c8e3b721702d008
paper_authors.name: Baoyuan Ou
paper_authors._id: 68385abc8c8e3b721702d009
paper_authors.name: Lai Xu
paper_authors._id: 68385abc8c8e3b721702d00a
paper_authors.name: Ruiwen Xu
paper_title: HoPE: Hybrid of Position Embedding for Length Generalization in
    Vision-Language Models
paper_summary: Vision-Language Models (VLMs) have made significant progress in multimodal
  tasks. However, their performance often deteriorates in long-context scenarios,
  particularly long videos. While Rotary Position Embedding (RoPE) has been
  widely adopted for length generalization in Large Language Models (LLMs),
  extending vanilla RoPE to capture the intricate spatial-temporal dependencies
  in videos remains an unsolved challenge. Existing methods typically allocate
  different frequencies within RoPE to encode 3D positional information. However,
  these allocation strategies mainly rely on heuristics, lacking in-depth
  theoretical analysis. In this paper, we first study how different allocation
  strategies impact the long-context capabilities of VLMs. Our analysis reveals
  that current multimodal RoPEs fail to reliably capture semantic similarities
  over extended contexts. To address this issue, we propose HoPE, a Hybrid of
  Position Embedding designed to improve the long-context capabilities of VLMs.
  HoPE introduces a hybrid frequency allocation strategy for reliable semantic
  modeling over arbitrarily long context, and a dynamic temporal scaling
  mechanism to facilitate robust learning and flexible inference across diverse
  context lengths. Extensive experiments across four video benchmarks on long
  video understanding and retrieval tasks demonstrate that HoPE consistently
  outperforms existing methods, confirming its effectiveness. Code is available
  at https://github.com/hrlics/HoPE.
paper_ai_summary: HoPE, a Hybrid of Position Embedding, enhances VLMs' long-context performance in videos through improved frequency allocation and dynamic temporal scaling.
paper_ai_keywords: Rotary Position Embedding RoPE Large Language Models LLMs multimodal RoPEs HoPE Hybrid of Position Embedding frequency allocation dynamic temporal scaling video benchmarks long video understanding retrieval tasks