InternVideo2: Scaling Video Foundation Models for Multimodal Video
  Understanding
We introduce InternVideo2, a new video foundation model (ViFM) that achieves
the state-of-the-art performance in action recognition, video-text tasks, and
video-centric dialogue. Our approach employs a progressive training paradigm
that unifies the different self- or weakly-supervised learning frameworks of
masked video token reconstruction, cross-modal contrastive learning, and next
token prediction. Different training stages would guide our model to capture
different levels of structure and semantic information through different
pretext tasks. At the data level, we prioritize the spatiotemporal consistency
by semantically segmenting videos and generating video-audio-speech captions.
This improves the alignment between video and text. We scale both data and
model size for our InternVideo2. Through extensive experiments, we validate our
designs and demonstrate the state-of-the-art performance on over 60 video and
audio tasks. Notably, our model outperforms others on various video-related
captioning, dialogue, and long video understanding benchmarks, highlighting its
ability to reason and comprehend long temporal contexts. Code and models are
available at https://github.com/OpenGVLab/InternVideo2/.
6600d5e3569b30694e4d1080
Yi Wang
6600d5e3569b30694e4d1081
Kunchang Li
admin_assigned
2024-03-25T10:05:21.549Z
61fb81006374891646732f37
https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
Kunchang Li
user
Andy1621
6600d5e3569b30694e4d1082
Xinhao Li
admin_assigned
2024-08-05T10:27:44.058Z
64171b10f1e86908935a493a
https://cdn-avatars.huggingface.co/v1/production/uploads/64171b10f1e86908935a493a/tAwbHgTw714Vz-Qudz6Ij.jpeg
Xinhao Li
user
lixinhao
6600d5e3569b30694e4d1083
Jiashuo Yu
admin_assigned
2024-03-25T10:05:49.550Z
64b89a14cf14c2fabe96664c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
Jiashuo Yu
user
awojustin
6600d5e3569b30694e4d1084
Yinan He
admin_assigned
2024-03-25T10:05:56.422Z
65b9d9961fe588f824fde191
/avatars/a9245958cc998a4b4b870bf2490fdaee.svg
Yinan He
user
yinanhe
6600d5e3569b30694e4d1085
Guo Chen
claimed_verified
2024-03-25T11:50:57.140Z
6392c73390b8e99a6779a7b0
/avatars/9ff824ab02848120aec5e8de6780bcf1.svg
Guo Chen
user
cg1177
6600d5e3569b30694e4d1086
Baoqi Pei
6600d5e3569b30694e4d1087
Rongkun Zheng
6600d5e3569b30694e4d1088
Jilan Xu
admin_assigned
2024-03-25T10:07:10.497Z
642e20d8018d9a57ca52af85
/avatars/c99c875a6654e7a8b3958777bcc46d8f.svg
Jilan Xu
user
Jazzcharles
6600d5e3569b30694e4d1089
Zun Wang
admin_assigned
2024-08-05T10:28:30.764Z
6622c710b0e5c5e3de8311c1
/avatars/a824c150040731679bbd77762ca9d4eb.svg
Zun Wang
user
ZunWang
6600d5e3569b30694e4d108a
Yansong Shi
admin_assigned
2024-08-05T10:28:36.354Z
66adeb38e990511973ae9a54
/avatars/bba250590937ef2ee46a05de55033fa8.svg
Yansong Shi
user
systemshi
6600d5e3569b30694e4d108b
Tianxiang Jiang
admin_assigned
2024-03-25T10:07:32.223Z
6502f42b2f87793060db18b4
/avatars/de12b6cd34a6149783fba39a947f4295.svg
Tianxiang Jiang
user
txjiang
6600d5e3569b30694e4d108c
Songze Li
claimed_verified
2025-07-03T07:47:46.457Z
64acbbd51aee69ece03c6c0c
/avatars/604df1cabc5faeda55022ae4c1997e56.svg
Songze Li
user
SongzeLi
6600d5e3569b30694e4d108d
Hongjie Zhang
6600d5e3569b30694e4d108e
Yifei Huang
6600d5e3569b30694e4d108f
Yu Qiao
6600d5e3569b30694e4d1090
Yali Wang
6600d5e3569b30694e4d1091
Limin Wang
admin_assigned
2024-03-25T10:09:26.539Z
62c77f4352d8ae531f5511f9
/avatars/50198ccb02ccd286975a4613fbabee28.svg
Limin Wang
user
lmwang
InternVideo2: Scaling Video Foundation Models for Multimodal Video
  Understanding
We introduce InternVideo2, a new video foundation model (ViFM) that achieves
the state-of-the-art performance in action recognition, video-text tasks, and
video-centric dialogue. Our approach employs a progressive training paradigm
that unifies the different self- or weakly-supervised learning frameworks of
masked video token reconstruction, cross-modal contrastive learning, and next
token prediction. Different training stages would guide our model to capture
different levels of structure and semantic information through different
pretext tasks. At the data level, we prioritize the spatiotemporal consistency
by semantically segmenting videos and generating video-audio-speech captions.
This improves the alignment between video and text. We scale both data and
model size for our InternVideo2. Through extensive experiments, we validate our
designs and demonstrate the state-of-the-art performance on over 60 video and
audio tasks. Notably, our model outperforms others on various video-related
captioning, dialogue, and long video understanding benchmarks, highlighting its
ability to reason and comprehend long temporal contexts. Code and models are
available at https://github.com/OpenGVLab/InternVideo2/.
InternVideo2, a state-of-the-art video foundation model, employs a progressive training approach using multiple self- or weakly-supervised learning paradigms to excel in various video and audio tasks.
video foundation model progressive training paradigm masked video token reconstruction cross-modal contrastive learning next token prediction spatiotemporal consistency semantic segmentation video-audio-speech captions long video understanding