title: InternVideo2: Scaling Video Foundation Models for Multimodal Video
    Understanding
summary: We introduce InternVideo2, a new video foundation model (ViFM) that achieves
  the state-of-the-art performance in action recognition, video-text tasks, and
  video-centric dialogue. Our approach employs a progressive training paradigm
  that unifies the different self- or weakly-supervised learning frameworks of
  masked video token reconstruction, cross-modal contrastive learning, and next
  token prediction. Different training stages would guide our model to capture
  different levels of structure and semantic information through different
  pretext tasks. At the data level, we prioritize the spatiotemporal consistency
  by semantically segmenting videos and generating video-audio-speech captions.
  This improves the alignment between video and text. We scale both data and
  model size for our InternVideo2. Through extensive experiments, we validate our
  designs and demonstrate the state-of-the-art performance on over 60 video and
  audio tasks. Notably, our model outperforms others on various video-related
  captioning, dialogue, and long video understanding benchmarks, highlighting its
  ability to reason and comprehend long temporal contexts. Code and models are
  available at https://github.com/OpenGVLab/InternVideo2/.
paper_authors._id: 6600d5e3569b30694e4d1080
paper_authors.name: Yi Wang
paper_authors._id: 6600d5e3569b30694e4d1081
paper_authors.name: Kunchang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:05:21.549Z
paper_authors._id: 61fb81006374891646732f37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
paper_authors.fullname: Kunchang Li
paper_authors.type: user
paper_authors.user: Andy1621
paper_authors._id: 6600d5e3569b30694e4d1082
paper_authors.name: Xinhao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T10:27:44.058Z
paper_authors._id: 64171b10f1e86908935a493a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64171b10f1e86908935a493a/tAwbHgTw714Vz-Qudz6Ij.jpeg
paper_authors.fullname: Xinhao Li
paper_authors.type: user
paper_authors.user: lixinhao
paper_authors._id: 6600d5e3569b30694e4d1083
paper_authors.name: Jiashuo Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:05:49.550Z
paper_authors._id: 64b89a14cf14c2fabe96664c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
paper_authors.fullname: Jiashuo Yu
paper_authors.type: user
paper_authors.user: awojustin
paper_authors._id: 6600d5e3569b30694e4d1084
paper_authors.name: Yinan He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:05:56.422Z
paper_authors._id: 65b9d9961fe588f824fde191
paper_authors.avatarUrl: /avatars/a9245958cc998a4b4b870bf2490fdaee.svg
paper_authors.fullname: Yinan He
paper_authors.type: user
paper_authors.user: yinanhe
paper_authors._id: 6600d5e3569b30694e4d1085
paper_authors.name: Guo Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-25T11:50:57.140Z
paper_authors._id: 6392c73390b8e99a6779a7b0
paper_authors.avatarUrl: /avatars/9ff824ab02848120aec5e8de6780bcf1.svg
paper_authors.fullname: Guo Chen
paper_authors.type: user
paper_authors.user: cg1177
paper_authors._id: 6600d5e3569b30694e4d1086
paper_authors.name: Baoqi Pei
paper_authors._id: 6600d5e3569b30694e4d1087
paper_authors.name: Rongkun Zheng
paper_authors._id: 6600d5e3569b30694e4d1088
paper_authors.name: Jilan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:07:10.497Z
paper_authors._id: 642e20d8018d9a57ca52af85
paper_authors.avatarUrl: /avatars/c99c875a6654e7a8b3958777bcc46d8f.svg
paper_authors.fullname: Jilan Xu
paper_authors.type: user
paper_authors.user: Jazzcharles
paper_authors._id: 6600d5e3569b30694e4d1089
paper_authors.name: Zun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T10:28:30.764Z
paper_authors._id: 6622c710b0e5c5e3de8311c1
paper_authors.avatarUrl: /avatars/a824c150040731679bbd77762ca9d4eb.svg
paper_authors.fullname: Zun Wang
paper_authors.type: user
paper_authors.user: ZunWang
paper_authors._id: 6600d5e3569b30694e4d108a
paper_authors.name: Yansong Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T10:28:36.354Z
paper_authors._id: 66adeb38e990511973ae9a54
paper_authors.avatarUrl: /avatars/bba250590937ef2ee46a05de55033fa8.svg
paper_authors.fullname: Yansong Shi
paper_authors.type: user
paper_authors.user: systemshi
paper_authors._id: 6600d5e3569b30694e4d108b
paper_authors.name: Tianxiang Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:07:32.223Z
paper_authors._id: 6502f42b2f87793060db18b4
paper_authors.avatarUrl: /avatars/de12b6cd34a6149783fba39a947f4295.svg
paper_authors.fullname: Tianxiang Jiang
paper_authors.type: user
paper_authors.user: txjiang
paper_authors._id: 6600d5e3569b30694e4d108c
paper_authors.name: Songze Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-05T10:28:46.687Z
paper_authors._id: 63e637b9b6a40bf941dd2513
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676031908262-noauth.jpeg
paper_authors.fullname: lisongze
paper_authors.type: user
paper_authors.user: Songze
paper_authors._id: 6600d5e3569b30694e4d108d
paper_authors.name: Hongjie Zhang
paper_authors._id: 6600d5e3569b30694e4d108e
paper_authors.name: Yifei Huang
paper_authors._id: 6600d5e3569b30694e4d108f
paper_authors.name: Yu Qiao
paper_authors._id: 6600d5e3569b30694e4d1090
paper_authors.name: Yali Wang
paper_authors._id: 6600d5e3569b30694e4d1091
paper_authors.name: Limin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:09:26.539Z
paper_authors._id: 62c77f4352d8ae531f5511f9
paper_authors.avatarUrl: /avatars/50198ccb02ccd286975a4613fbabee28.svg
paper_authors.fullname: Limin Wang
paper_authors.type: user
paper_authors.user: lmwang
paper_title: InternVideo2: Scaling Video Foundation Models for Multimodal Video
    Understanding
paper_summary: We introduce InternVideo2, a new video foundation model (ViFM) that achieves
  the state-of-the-art performance in action recognition, video-text tasks, and
  video-centric dialogue. Our approach employs a progressive training paradigm
  that unifies the different self- or weakly-supervised learning frameworks of
  masked video token reconstruction, cross-modal contrastive learning, and next
  token prediction. Different training stages would guide our model to capture
  different levels of structure and semantic information through different
  pretext tasks. At the data level, we prioritize the spatiotemporal consistency
  by semantically segmenting videos and generating video-audio-speech captions.
  This improves the alignment between video and text. We scale both data and
  model size for our InternVideo2. Through extensive experiments, we validate our
  designs and demonstrate the state-of-the-art performance on over 60 video and
  audio tasks. Notably, our model outperforms others on various video-related
  captioning, dialogue, and long video understanding benchmarks, highlighting its
  ability to reason and comprehend long temporal contexts. Code and models are
  available at https://github.com/OpenGVLab/InternVideo2/.
paper_ai_summary: InternVideo2, a state-of-the-art video foundation model, employs a progressive training approach using multiple self- or weakly-supervised learning paradigms to excel in various video and audio tasks.
paper_ai_keywords: video foundation model progressive training paradigm masked video token reconstruction cross-modal contrastive learning next token prediction spatiotemporal consistency semantic segmentation video-audio-speech captions long video understanding