title: Long Context Transfer from Language to Vision
summary: Video sequences offer valuable temporal information, but existing large
  multimodal models (LMMs) fall short in understanding extremely long videos.
  Many works address this by reducing the number of visual tokens using visual
  resamplers. Alternatively, in this paper, we approach this problem from the
  perspective of the language model. By simply extrapolating the context length
  of the language backbone, we enable LMMs to comprehend orders of magnitude more
  visual tokens without any video training. We call this phenomenon long context
  transfer and carefully ablate its properties. To effectively measure LMMs'
  ability to generalize to long contexts in the vision modality, we develop
  V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark
  inspired by the language model's NIAH test. Our proposed Long Video Assistant
  (LongVA) can process 2000 frames or over 200K visual tokens without additional
  complexities. With its extended context length, LongVA achieves
  state-of-the-art performance on Video-MME among 7B-scale models by densely
  sampling more input frames. Our work is open-sourced at
  https://github.com/EvolvingLMMs-Lab/LongVA.
paper_authors._id: 667a3a30b032d9a6594c8c99
paper_authors.name: Peiyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:33:11.134Z
paper_authors._id: 63565cc56d7fcf1bedb7d347
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
paper_authors.fullname: Zhang Peiyuan
paper_authors.type: user
paper_authors.user: PY007
paper_authors._id: 667a3a30b032d9a6594c8c9a
paper_authors.name: Kaichen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:33:08.902Z
paper_authors._id: 64bb77e786e7fb5b8a317a43
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64bb77e786e7fb5b8a317a43/J0jOrlZJ9gazdYaeSH2Bo.png
paper_authors.fullname: kcz
paper_authors.type: user
paper_authors.user: kcz358
paper_authors._id: 667a3a30b032d9a6594c8c9b
paper_authors.name: Bo Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:48:50.584Z
paper_authors._id: 62d3f7d84b0933c48f3cdd9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: luodian
paper_authors._id: 667a3a30b032d9a6594c8c9c
paper_authors.name: Guangtao Zeng
paper_authors._id: 667a3a30b032d9a6594c8c9d
paper_authors.name: Jingkang Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:07:05.709Z
paper_authors._id: 62b5777f593a2c49da69dc02
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658152070753-62b5777f593a2c49da69dc02.jpeg
paper_authors.fullname: Jingkang Yang
paper_authors.type: user
paper_authors.user: Jingkang
paper_authors._id: 667a3a30b032d9a6594c8c9e
paper_authors.name: Yuanhan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:07:43.895Z
paper_authors._id: 62a993d80472c0b7f94027df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
paper_authors.fullname: Zhang Yuanhan
paper_authors.type: user
paper_authors.user: ZhangYuanhan
paper_authors._id: 667a3a30b032d9a6594c8c9f
paper_authors.name: Ziyue Wang
paper_authors._id: 667a3a30b032d9a6594c8ca0
paper_authors.name: Haoran Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:05:03.718Z
paper_authors._id: 652802a42949e6071c703550
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652802a42949e6071c703550/dQF73cLlAoSLtgi8JzbnI.jpeg
paper_authors.fullname: Haoran Tan
paper_authors.type: user
paper_authors.user: Tan0118
paper_authors._id: 667a3a30b032d9a6594c8ca1
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:09:27.187Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 667a3a30b032d9a6594c8ca2
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:09:40.723Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: Long Context Transfer from Language to Vision
paper_summary: Video sequences offer valuable temporal information, but existing large
  multimodal models (LMMs) fall short in understanding extremely long videos.
  Many works address this by reducing the number of visual tokens using visual
  resamplers. Alternatively, in this paper, we approach this problem from the
  perspective of the language model. By simply extrapolating the context length
  of the language backbone, we enable LMMs to comprehend orders of magnitude more
  visual tokens without any video training. We call this phenomenon long context
  transfer and carefully ablate its properties. To effectively measure LMMs'
  ability to generalize to long contexts in the vision modality, we develop
  V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark
  inspired by the language model's NIAH test. Our proposed Long Video Assistant
  (LongVA) can process 2000 frames or over 200K visual tokens without additional
  complexities. With its extended context length, LongVA achieves
  state-of-the-art performance on Video-MME among 7B-scale models by densely
  sampling more input frames. Our work is open-sourced at
  https://github.com/EvolvingLMMs-Lab/LongVA.
paper_ai_summary: By extending the context length of the language backbone, LMMs can process extremely long video sequences, achieving state-of-the-art performance without additional training or complexity.
paper_ai_keywords: visual resamplers language backbone long context transfer V-NIAH Long Video Assistant Video-MME dense sampling