title: LongVILA: Scaling Long-Context Visual Language Models for Long Videos
summary: Long-context capability is critical for multi-modal foundation models. We
  introduce LongVILA, a full-stack solution for long-context vision-language
  models, including system, model training, and dataset development. On the
  system side, we introduce the first Multi-Modal Sequence Parallelism (MM-SP)
  system that enables long-context training and inference, enabling 2M context
  length training on 256 GPUs. MM-SP is also efficient, being 2.1x - 5.7x faster
  than Ring-Style Sequence Parallelism and 1.1x - 1.4x faster than Megatron-LM in
  text-only settings. Moreover, it seamlessly integrates with Hugging Face
  Transformers. For model training, we propose a five-stage pipeline comprising
  alignment, pre-training, context extension, and long-short joint supervised
  fine-tuning. Regarding datasets, we meticulously construct large-scale visual
  language pre-training datasets and long video instruction-following datasets to
  support our multi-stage training process. The full-stack solution extends the
  feasible frame number of VILA by a factor of 128 (from 8 to 1024 frames) and
  improves long video captioning score from 2.00 to 3.26 (1.6x), achieving 99.5%
  accuracy in 1400-frames video (274k context length) needle in a haystack.
  LongVILA-8B also demonstrates a consistent improvement in performance on long
  videos within the VideoMME benchmark as the video frames increase.
paper_authors._id: 66c41182ea476bea05da1a4c
paper_authors.name: Fuzhao Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T16:25:05.182Z
paper_authors._id: 64a52314978242bb77e05790
paper_authors.avatarUrl: /avatars/e97206c653de3446266c80c2943c0a6f.svg
paper_authors.fullname: Fuzhao Xue
paper_authors.type: user
paper_authors.user: fuzhao
paper_authors._id: 66c41182ea476bea05da1a4d
paper_authors.name: Yukang Chen
paper_authors._id: 66c41182ea476bea05da1a4e
paper_authors.name: Dacheng Li
paper_authors._id: 66c41182ea476bea05da1a4f
paper_authors.name: Qinghao Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T06:57:25.574Z
paper_authors._id: 62fb8412610dae1bcd036db7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62fb8412610dae1bcd036db7/LIaEoDrvrq8hDl8509ZuF.jpeg
paper_authors.fullname: Qinghao Hu
paper_authors.type: user
paper_authors.user: Qinghao
paper_authors._id: 66c41182ea476bea05da1a50
paper_authors.name: Ligeng Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:39:35.010Z
paper_authors._id: 62b4b5beb25cb80fcf278354
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b4b5beb25cb80fcf278354/7uGNOeE91paZikhK2QFgp.jpeg
paper_authors.fullname: Ligeng Zhu
paper_authors.type: user
paper_authors.user: Ligeng-Zhu
paper_authors._id: 66c41182ea476bea05da1a51
paper_authors.name: Xiuyu Li
paper_authors._id: 66c41182ea476bea05da1a52
paper_authors.name: Yunhao Fang
paper_authors._id: 66c41182ea476bea05da1a53
paper_authors.name: Haotian Tang
paper_authors._id: 66c41182ea476bea05da1a54
paper_authors.name: Shang Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T10:00:46.154Z
paper_authors._id: 641d8bacd526196afc12766d
paper_authors.avatarUrl: /avatars/73f7b2d86a7bf27940bec2b1f199d71b.svg
paper_authors.fullname: Shang Yang
paper_authors.type: user
paper_authors.user: Shangy
paper_authors._id: 66c41182ea476bea05da1a55
paper_authors.name: Zhijian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:59:36.699Z
paper_authors._id: 650dac79b959b0e1d41d7378
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
paper_authors.fullname: Zhijian Liu
paper_authors.type: user
paper_authors.user: zhijianliu
paper_authors._id: 66c41182ea476bea05da1a56
paper_authors.name: Ethan He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:28:50.068Z
paper_authors._id: 62b20f6df4a72794189248fc
paper_authors.avatarUrl: /avatars/87e1125868616d4f7d6ee1e5ec4499b4.svg
paper_authors.fullname: Ethan He
paper_authors.type: user
paper_authors.user: ethanhe
paper_authors._id: 66c41182ea476bea05da1a57
paper_authors.name: Hongxu Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-27T09:06:10.536Z
paper_authors._id: 65a8b7f69aec1645994e7a15
paper_authors.avatarUrl: /avatars/debc086f3fea029db22847bde80799a0.svg
paper_authors.fullname: Hongxu Yin
paper_authors.type: user
paper_authors.user: yinhongxu
paper_authors._id: 66c41182ea476bea05da1a58
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:07:48.737Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_authors._id: 66c41182ea476bea05da1a59
paper_authors.name: Jan Kautz
paper_authors._id: 66c41182ea476bea05da1a5a
paper_authors.name: Linxi Fan
paper_authors._id: 66c41182ea476bea05da1a5b
paper_authors.name: Yuke Zhu
paper_authors._id: 66c41182ea476bea05da1a5c
paper_authors.name: Yao Lu
paper_authors._id: 66c41182ea476bea05da1a5d
paper_authors.name: Song Han
paper_title: LongVILA: Scaling Long-Context Visual Language Models for Long Videos
paper_summary: Long-context capability is critical for multi-modal foundation models. We
  introduce LongVILA, a full-stack solution for long-context vision-language
  models, including system, model training, and dataset development. On the
  system side, we introduce the first Multi-Modal Sequence Parallelism (MM-SP)
  system that enables long-context training and inference, enabling 2M context
  length training on 256 GPUs. MM-SP is also efficient, being 2.1x - 5.7x faster
  than Ring-Style Sequence Parallelism and 1.1x - 1.4x faster than Megatron-LM in
  text-only settings. Moreover, it seamlessly integrates with Hugging Face
  Transformers. For model training, we propose a five-stage pipeline comprising
  alignment, pre-training, context extension, and long-short joint supervised
  fine-tuning. Regarding datasets, we meticulously construct large-scale visual
  language pre-training datasets and long video instruction-following datasets to
  support our multi-stage training process. The full-stack solution extends the
  feasible frame number of VILA by a factor of 128 (from 8 to 1024 frames) and
  improves long video captioning score from 2.00 to 3.26 (1.6x), achieving 99.5%
  accuracy in 1400-frames video (274k context length) needle in a haystack.
  LongVILA-8B also demonstrates a consistent improvement in performance on long
  videos within the VideoMME benchmark as the video frames increase.
paper_ai_summary: LongVILA, a full-stack solution for long-context vision-language models, introduces Multi-Modal Sequence Parallelism for efficient training and inference, and a five-stage training pipeline that enhances long video processing and captioning performance.
paper_ai_keywords: Multi-Modal Sequence Parallelism MM-SP long-context vision-language models alignment pre-training context extension long-short joint supervised fine-tuning VideoMME benchmark