Long Context Transfer from Language to Vision
Video sequences offer valuable temporal information, but existing large
multimodal models (LMMs) fall short in understanding extremely long videos.
Many works address this by reducing the number of visual tokens using visual
resamplers. Alternatively, in this paper, we approach this problem from the
perspective of the language model. By simply extrapolating the context length
of the language backbone, we enable LMMs to comprehend orders of magnitude more
visual tokens without any video training. We call this phenomenon long context
transfer and carefully ablate its properties. To effectively measure LMMs'
ability to generalize to long contexts in the vision modality, we develop
V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark
inspired by the language model's NIAH test. Our proposed Long Video Assistant
(LongVA) can process 2000 frames or over 200K visual tokens without additional
complexities. With its extended context length, LongVA achieves
state-of-the-art performance on Video-MME among 7B-scale models by densely
sampling more input frames. Our work is open-sourced at
https://github.com/EvolvingLMMs-Lab/LongVA.
667a3a30b032d9a6594c8c99
Peiyuan Zhang
claimed_verified
2024-06-25T07:33:11.134Z
63565cc56d7fcf1bedb7d347
https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
Zhang Peiyuan
user
PY007
667a3a30b032d9a6594c8c9a
Kaichen Zhang
claimed_verified
2024-06-25T07:33:08.902Z
64bb77e786e7fb5b8a317a43
https://cdn-avatars.huggingface.co/v1/production/uploads/64bb77e786e7fb5b8a317a43/J0jOrlZJ9gazdYaeSH2Bo.png
kcz
user
kcz358
667a3a30b032d9a6594c8c9b
Bo Li
claimed_verified
2024-08-09T07:48:50.584Z
62d3f7d84b0933c48f3cdd9c
https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
Bo Li
user
luodian
667a3a30b032d9a6594c8c9c
Guangtao Zeng
667a3a30b032d9a6594c8c9d
Jingkang Yang
admin_assigned
2024-06-25T08:07:05.709Z
62b5777f593a2c49da69dc02
https://cdn-avatars.huggingface.co/v1/production/uploads/1658152070753-62b5777f593a2c49da69dc02.jpeg
Jingkang Yang
user
Jingkang
667a3a30b032d9a6594c8c9e
Yuanhan Zhang
admin_assigned
2024-06-25T08:07:43.895Z
62a993d80472c0b7f94027df
https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
Zhang Yuanhan
user
ZhangYuanhan
667a3a30b032d9a6594c8c9f
Ziyue Wang
667a3a30b032d9a6594c8ca0
Haoran Tan
claimed_verified
2024-06-26T08:05:03.718Z
652802a42949e6071c703550
https://cdn-avatars.huggingface.co/v1/production/uploads/652802a42949e6071c703550/dQF73cLlAoSLtgi8JzbnI.jpeg
Haoran Tan
user
Tan0118
667a3a30b032d9a6594c8ca1
Chunyuan Li
admin_assigned
2024-06-25T08:09:27.187Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
667a3a30b032d9a6594c8ca2
Ziwei Liu
admin_assigned
2024-06-25T08:09:40.723Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
Long Context Transfer from Language to Vision
Video sequences offer valuable temporal information, but existing large
multimodal models (LMMs) fall short in understanding extremely long videos.
Many works address this by reducing the number of visual tokens using visual
resamplers. Alternatively, in this paper, we approach this problem from the
perspective of the language model. By simply extrapolating the context length
of the language backbone, we enable LMMs to comprehend orders of magnitude more
visual tokens without any video training. We call this phenomenon long context
transfer and carefully ablate its properties. To effectively measure LMMs'
ability to generalize to long contexts in the vision modality, we develop
V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark
inspired by the language model's NIAH test. Our proposed Long Video Assistant
(LongVA) can process 2000 frames or over 200K visual tokens without additional
complexities. With its extended context length, LongVA achieves
state-of-the-art performance on Video-MME among 7B-scale models by densely
sampling more input frames. Our work is open-sourced at
https://github.com/EvolvingLMMs-Lab/LongVA.
By extending the context length of the language backbone, LMMs can process extremely long video sequences, achieving state-of-the-art performance without additional training or complexity.
visual resamplers language backbone long context transfer V-NIAH Long Video Assistant Video-MME dense sampling