VideoRoPE: What Makes for Good Video Rotary Position Embedding?
While Rotary Position Embedding (RoPE) and its variants are widely adopted
for their long-context capabilities, the extension of the 1D RoPE to video,
with its complex spatio-temporal structure, remains an open challenge. This
work first introduces a comprehensive analysis that identifies four key
characteristics essential for the effective adaptation of RoPE to video, which
have not been fully considered in prior work. As part of our analysis, we
introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors)
task, which adds periodic distractors into V-NIAH. The V-NIAH-D task
demonstrates that previous RoPE variants, lacking appropriate temporal
dimension allocation, are easily misled by distractors. Based on our analysis,
we introduce VideoRoPE, with a 3D structure designed to
preserve spatio-temporal relationships. VideoRoPE features
low-frequency temporal allocation to mitigate periodic oscillations, a
diagonal layout to maintain spatial symmetry, and adjustable
temporal spacing to decouple temporal and spatial indexing. VideoRoPE
consistently surpasses previous RoPE variants, across diverse downstream tasks
such as long video retrieval, video understanding, and video hallucination. Our
code will be available at
https://github.com/Wiselnn570/VideoRoPE{https://github.com/Wiselnn570/VideoRoPE}.
67a97a47174028234b74f687
Xilin Wei
claimed_verified
2025-02-10T13:12:02.432Z
62eb70462f0f5e54df42f778
/avatars/456049dba67638d3cdb330cdf383f272.svg
Xilin Wei
user
Wiselnn
67a97a47174028234b74f688
Xiaoran Liu
claimed_verified
2025-02-10T09:49:59.999Z
64f033ef82c6eea604c4da8b
/avatars/51b93fea7fd68b4274ee03701245dcca.svg
Liu Xiaoran
user
LiuXR
67a97a47174028234b74f689
Yuhang Zang
claimed_verified
2025-02-10T09:50:02.011Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
67a97a47174028234b74f68a
Xiaoyi Dong
67a97a47174028234b74f68b
Pan Zhang
67a97a47174028234b74f68c
Yuhang Cao
admin_assigned
2025-02-10T16:01:29.622Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
67a97a47174028234b74f68d
Jian Tong
67a97a47174028234b74f68e
Haodong Duan
admin_assigned
2025-02-10T16:01:41.685Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
67a97a47174028234b74f68f
Qipeng Guo
admin_assigned
2025-02-10T16:01:48.883Z
6491cd52b1e5d3444528edb1
/avatars/a85635d886c7f157b6723dec5c01c030.svg
Qipeng Guo
user
QipengGuo
67a97a47174028234b74f690
Jiaqi Wang
claimed_verified
2025-05-21T09:09:11.292Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
67a97a47174028234b74f691
Xipeng Qiu
admin_assigned
2025-02-10T16:02:04.834Z
61457b8deff2c9fdb4de4988
https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
Xipeng Qiu
user
xpqiu
67a97a47174028234b74f692
Dahua Lin
admin_assigned
2025-02-10T16:02:10.781Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
VideoRoPE: What Makes for Good Video Rotary Position Embedding?
While Rotary Position Embedding (RoPE) and its variants are widely adopted
for their long-context capabilities, the extension of the 1D RoPE to video,
with its complex spatio-temporal structure, remains an open challenge. This
work first introduces a comprehensive analysis that identifies four key
characteristics essential for the effective adaptation of RoPE to video, which
have not been fully considered in prior work. As part of our analysis, we
introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors)
task, which adds periodic distractors into V-NIAH. The V-NIAH-D task
demonstrates that previous RoPE variants, lacking appropriate temporal
dimension allocation, are easily misled by distractors. Based on our analysis,
we introduce VideoRoPE, with a 3D structure designed to
preserve spatio-temporal relationships. VideoRoPE features
low-frequency temporal allocation to mitigate periodic oscillations, a
diagonal layout to maintain spatial symmetry, and adjustable
temporal spacing to decouple temporal and spatial indexing. VideoRoPE
consistently surpasses previous RoPE variants, across diverse downstream tasks
such as long video retrieval, video understanding, and video hallucination. Our
code will be available at
https://github.com/Wiselnn570/VideoRoPE{https://github.com/Wiselnn570/VideoRoPE}.
VideoRoPE, a 3D position embedding that preserves spatio-temporal relationships, improves performance on video tasks by addressing limitations in previous RoPE variants.
Rotary Position Embedding RoPE long-context capabilities V-NIAH-D diagonal layout low-frequency temporal allocation adjustable temporal spacing long video retrieval video understanding video hallucination