title: VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video
    Understanding
summary: In this paper, we propose VideoLLaMA3, a more advanced multimodal foundation
  model for image and video understanding. The core design philosophy of
  VideoLLaMA3 is vision-centric. The meaning of "vision-centric" is two-fold: the
  vision-centric training paradigm and vision-centric framework design. The key
  insight of our vision-centric training paradigm is that high-quality image-text
  data is crucial for both image and video understanding. Instead of preparing
  massive video-text datasets, we focus on constructing large-scale and
  high-quality image-text datasets. VideoLLaMA3 has four training stages: 1)
  vision-centric alignment stage, which warms up the vision encoder and
  projector; 2) vision-language pretraining stage, which jointly tunes the vision
  encoder, projector, and LLM with large-scale image-text data covering multiple
  types (including scene images, documents, charts) as well as text-only data. 3)
  multi-task fine-tuning stage, which incorporates image-text SFT data for
  downstream tasks and video-text data to establish a foundation for video
  understanding. 4) video-centric fine-tuning, which further improves the model's
  capability in video understanding. As for the framework design, to better
  capture fine-grained details in images, the pretrained vision encoder is
  adapted to encode images of varying sizes into vision tokens with corresponding
  numbers, rather than a fixed number of tokens. For video inputs, we reduce the
  number of vision tokens according to their similarity so that the
  representation of videos will be more precise and compact. Benefit from
  vision-centric designs, VideoLLaMA3 achieves compelling performances in both
  image and video understanding benchmarks.
paper_authors._id: 6791cced89c55efdc9fd45e5
paper_authors.name: Boqiang Zhang
paper_authors._id: 6791cced89c55efdc9fd45e6
paper_authors.name: Kehan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:38:16.478Z
paper_authors._id: 66836aba7b50b433cd681182
paper_authors.avatarUrl: /avatars/483a6fef6d27680d19f9ea1789122bdd.svg
paper_authors.fullname: Kehan LI
paper_authors.type: user
paper_authors.user: CausalLi
paper_authors._id: 6791cced89c55efdc9fd45e7
paper_authors.name: Zesen Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T09:17:49.716Z
paper_authors._id: 65b2529285b6c21448a10d65
paper_authors.avatarUrl: /avatars/1b09e2742aecce1bbdc57f0c4504cf38.svg
paper_authors.fullname: Zesen Cheng
paper_authors.type: user
paper_authors.user: ClownRat
paper_authors._id: 6791cced89c55efdc9fd45e8
paper_authors.name: Zhiqiang Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T09:17:47.840Z
paper_authors._id: 637f228152229c63921119c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f228152229c63921119c3/acwXorra1r9_7i3KlBFjS.jpeg
paper_authors.fullname: Zhiqiang Hu
paper_authors.type: user
paper_authors.user: Zhiqiang007
paper_authors._id: 6791cced89c55efdc9fd45e9
paper_authors.name: Yuqian Yuan
paper_authors._id: 6791cced89c55efdc9fd45ea
paper_authors.name: Guanzheng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:38:44.522Z
paper_authors._id: 645475e2548f22be59847604
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645475e2548f22be59847604/EhSurrZ25u31qQ2TVXQXt.jpeg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Guanzheng
paper_authors._id: 6791cced89c55efdc9fd45eb
paper_authors.name: Sicong Leng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:38:54.047Z
paper_authors._id: 609115c79a8bcaa437b234a9
paper_authors.avatarUrl: /avatars/1631a91030703d8397133363cf82c863.svg
paper_authors.fullname: Leng Sicong
paper_authors.type: user
paper_authors.user: Sicong
paper_authors._id: 6791cced89c55efdc9fd45ec
paper_authors.name: Yuming Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:39:08.190Z
paper_authors._id: 629c95b7a5d6f5fe10e6ed45
paper_authors.avatarUrl: /avatars/c9a5f0e7fae3a079d26db266e9ff90a3.svg
paper_authors.fullname: Yuming Jiang
paper_authors.type: user
paper_authors.user: yumingj
paper_authors._id: 6791cced89c55efdc9fd45ed
paper_authors.name: Hang Zhang
paper_authors._id: 6791cced89c55efdc9fd45ee
paper_authors.name: Xin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-07T09:58:35.000Z
paper_authors._id: 63913b120cf6b11c487ca31d
paper_authors.avatarUrl: /avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
paper_authors.fullname: Xin Li
paper_authors.type: user
paper_authors.user: lixin4ever
paper_authors._id: 6791cced89c55efdc9fd45ef
paper_authors.name: Peng Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T15:05:03.480Z
paper_authors._id: 651585ea18cfed0d30bee586
paper_authors.avatarUrl: /avatars/579e468334102472d870875fe40302e6.svg
paper_authors.fullname: Peng Jin
paper_authors.type: user
paper_authors.user: Chat-UniVi
paper_authors._id: 6791cced89c55efdc9fd45f0
paper_authors.name: Wenqi Zhang
paper_authors._id: 6791cced89c55efdc9fd45f1
paper_authors.name: Fan Wang
paper_authors._id: 6791cced89c55efdc9fd45f2
paper_authors.name: Lidong Bing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-23T09:39:36.775Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_authors._id: 6791cced89c55efdc9fd45f3
paper_authors.name: Deli Zhao
paper_title: VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video
    Understanding
paper_summary: In this paper, we propose VideoLLaMA3, a more advanced multimodal foundation
  model for image and video understanding. The core design philosophy of
  VideoLLaMA3 is vision-centric. The meaning of "vision-centric" is two-fold: the
  vision-centric training paradigm and vision-centric framework design. The key
  insight of our vision-centric training paradigm is that high-quality image-text
  data is crucial for both image and video understanding. Instead of preparing
  massive video-text datasets, we focus on constructing large-scale and
  high-quality image-text datasets. VideoLLaMA3 has four training stages: 1)
  vision-centric alignment stage, which warms up the vision encoder and
  projector; 2) vision-language pretraining stage, which jointly tunes the vision
  encoder, projector, and LLM with large-scale image-text data covering multiple
  types (including scene images, documents, charts) as well as text-only data. 3)
  multi-task fine-tuning stage, which incorporates image-text SFT data for
  downstream tasks and video-text data to establish a foundation for video
  understanding. 4) video-centric fine-tuning, which further improves the model's
  capability in video understanding. As for the framework design, to better
  capture fine-grained details in images, the pretrained vision encoder is
  adapted to encode images of varying sizes into vision tokens with corresponding
  numbers, rather than a fixed number of tokens. For video inputs, we reduce the
  number of vision tokens according to their similarity so that the
  representation of videos will be more precise and compact. Benefit from
  vision-centric designs, VideoLLaMA3 achieves compelling performances in both
  image and video understanding benchmarks.
paper_ai_summary: A vision-centric multimodal foundation model, VideoLLaMA3, improves image and video understanding through strategic training stages and adaptive vision token encoding.
paper_ai_keywords: vision-centric training paradigm vision-centric framework design vision-centric alignment stage vision-language pretraining stage multi-task fine-tuning stage video-centric fine-tuning vision tokens adaptive vision encoder