VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video
  Understanding
In this paper, we propose VideoLLaMA3, a more advanced multimodal foundation
model for image and video understanding. The core design philosophy of
VideoLLaMA3 is vision-centric. The meaning of "vision-centric" is two-fold: the
vision-centric training paradigm and vision-centric framework design. The key
insight of our vision-centric training paradigm is that high-quality image-text
data is crucial for both image and video understanding. Instead of preparing
massive video-text datasets, we focus on constructing large-scale and
high-quality image-text datasets. VideoLLaMA3 has four training stages: 1)
vision-centric alignment stage, which warms up the vision encoder and
projector; 2) vision-language pretraining stage, which jointly tunes the vision
encoder, projector, and LLM with large-scale image-text data covering multiple
types (including scene images, documents, charts) as well as text-only data. 3)
multi-task fine-tuning stage, which incorporates image-text SFT data for
downstream tasks and video-text data to establish a foundation for video
understanding. 4) video-centric fine-tuning, which further improves the model's
capability in video understanding. As for the framework design, to better
capture fine-grained details in images, the pretrained vision encoder is
adapted to encode images of varying sizes into vision tokens with corresponding
numbers, rather than a fixed number of tokens. For video inputs, we reduce the
number of vision tokens according to their similarity so that the
representation of videos will be more precise and compact. Benefit from
vision-centric designs, VideoLLaMA3 achieves compelling performances in both
image and video understanding benchmarks.
6791cced89c55efdc9fd45e5
Boqiang Zhang
6791cced89c55efdc9fd45e6
Kehan Li
admin_assigned
2025-01-23T09:38:16.478Z
66836aba7b50b433cd681182
/avatars/483a6fef6d27680d19f9ea1789122bdd.svg
Kehan LI
user
CausalLi
6791cced89c55efdc9fd45e7
Zesen Cheng
claimed_verified
2025-01-23T09:17:49.716Z
65b2529285b6c21448a10d65
/avatars/1b09e2742aecce1bbdc57f0c4504cf38.svg
Zesen Cheng
user
ClownRat
6791cced89c55efdc9fd45e8
Zhiqiang Hu
claimed_verified
2025-01-23T09:17:47.840Z
637f228152229c63921119c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637f228152229c63921119c3/acwXorra1r9_7i3KlBFjS.jpeg
Zhiqiang Hu
user
Zhiqiang007
6791cced89c55efdc9fd45e9
Yuqian Yuan
6791cced89c55efdc9fd45ea
Guanzheng Chen
admin_assigned
2025-01-23T09:38:44.522Z
645475e2548f22be59847604
https://cdn-avatars.huggingface.co/v1/production/uploads/645475e2548f22be59847604/EhSurrZ25u31qQ2TVXQXt.jpeg
Chen
user
Guanzheng
6791cced89c55efdc9fd45eb
Sicong Leng
admin_assigned
2025-01-23T09:38:54.047Z
609115c79a8bcaa437b234a9
/avatars/1631a91030703d8397133363cf82c863.svg
Leng Sicong
user
Sicong
6791cced89c55efdc9fd45ec
Yuming Jiang
admin_assigned
2025-01-23T09:39:08.190Z
629c95b7a5d6f5fe10e6ed45
/avatars/c9a5f0e7fae3a079d26db266e9ff90a3.svg
Yuming Jiang
user
yumingj
6791cced89c55efdc9fd45ed
Hang Zhang
6791cced89c55efdc9fd45ee
Xin Li
claimed_verified
2025-02-07T09:58:35.000Z
63913b120cf6b11c487ca31d
/avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
Xin Li
user
lixin4ever
6791cced89c55efdc9fd45ef
Peng Jin
claimed_verified
2025-01-23T15:05:03.480Z
651585ea18cfed0d30bee586
/avatars/579e468334102472d870875fe40302e6.svg
Peng Jin
user
Chat-UniVi
6791cced89c55efdc9fd45f0
Wenqi Zhang
6791cced89c55efdc9fd45f1
Fan Wang
6791cced89c55efdc9fd45f2
Lidong Bing
admin_assigned
2025-01-23T09:39:36.775Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
6791cced89c55efdc9fd45f3
Deli Zhao
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video
  Understanding
In this paper, we propose VideoLLaMA3, a more advanced multimodal foundation
model for image and video understanding. The core design philosophy of
VideoLLaMA3 is vision-centric. The meaning of "vision-centric" is two-fold: the
vision-centric training paradigm and vision-centric framework design. The key
insight of our vision-centric training paradigm is that high-quality image-text
data is crucial for both image and video understanding. Instead of preparing
massive video-text datasets, we focus on constructing large-scale and
high-quality image-text datasets. VideoLLaMA3 has four training stages: 1)
vision-centric alignment stage, which warms up the vision encoder and
projector; 2) vision-language pretraining stage, which jointly tunes the vision
encoder, projector, and LLM with large-scale image-text data covering multiple
types (including scene images, documents, charts) as well as text-only data. 3)
multi-task fine-tuning stage, which incorporates image-text SFT data for
downstream tasks and video-text data to establish a foundation for video
understanding. 4) video-centric fine-tuning, which further improves the model's
capability in video understanding. As for the framework design, to better
capture fine-grained details in images, the pretrained vision encoder is
adapted to encode images of varying sizes into vision tokens with corresponding
numbers, rather than a fixed number of tokens. For video inputs, we reduce the
number of vision tokens according to their similarity so that the
representation of videos will be more precise and compact. Benefit from
vision-centric designs, VideoLLaMA3 achieves compelling performances in both
image and video understanding benchmarks.
A vision-centric multimodal foundation model, VideoLLaMA3, improves image and video understanding through strategic training stages and adaptive vision token encoding.
vision-centric training paradigm vision-centric framework design vision-centric alignment stage vision-language pretraining stage multi-task fine-tuning stage video-centric fine-tuning vision tokens adaptive vision encoder