title: VideoChat: Chat-Centric Video Understanding
summary: In this study, we initiate an exploration into video understanding by
  introducing VideoChat, an end-to-end chat-centric video understanding system.
  It integrates video foundation models and large language models via a learnable
  neural interface, excelling in spatiotemporal reasoning, event localization,
  and causal relationship inference. To instructively tune this system, we
  propose a video-centric instruction dataset, composed of thousands of videos
  matched with detailed descriptions and conversations. This dataset emphasizes
  spatiotemporal reasoning and causal relationships, providing a valuable asset
  for training chat-centric video understanding systems. Preliminary qualitative
  experiments reveal our system's potential across a broad spectrum of video
  applications and set the standard for future research. Access our code and data
  at https://github.com/OpenGVLab/Ask-Anything
paper_authors._id: 645c4d10a7db8b98eb397d1e
paper_authors.name: KunChang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:48:06.848Z
paper_authors._id: 61fb81006374891646732f37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
paper_authors.fullname: Kunchang Li
paper_authors.type: user
paper_authors.user: Andy1621
paper_authors._id: 645c4d10a7db8b98eb397d1f
paper_authors.name: Yinan He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-17T06:49:29.393Z
paper_authors._id: 62aafa49f29ff279b51f0182
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
paper_authors.fullname: yinanhe
paper_authors.type: user
paper_authors.user: ynhe
paper_authors._id: 645c4d10a7db8b98eb397d20
paper_authors.name: Yi Wang
paper_authors._id: 645c4d10a7db8b98eb397d21
paper_authors.name: Yizhuo Li
paper_authors._id: 645c4d10a7db8b98eb397d22
paper_authors.name: Wenhai Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:48:54.344Z
paper_authors._id: 644f58e6f8b353c94914c7e0
paper_authors.avatarUrl: /avatars/16c7e80aafbea9a52cfc309068eedc9a.svg
paper_authors.fullname: wangwenhai
paper_authors.type: user
paper_authors.user: yicixi
paper_authors._id: 645c4d10a7db8b98eb397d23
paper_authors.name: Ping Luo
paper_authors._id: 645c4d10a7db8b98eb397d24
paper_authors.name: Yali Wang
paper_authors._id: 645c4d10a7db8b98eb397d25
paper_authors.name: Limin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:49:37.149Z
paper_authors._id: 62c77f4352d8ae531f5511f9
paper_authors.avatarUrl: /avatars/50198ccb02ccd286975a4613fbabee28.svg
paper_authors.fullname: Limin Wang
paper_authors.type: user
paper_authors.user: lmwang
paper_authors._id: 645c4d10a7db8b98eb397d26
paper_authors.name: Yu Qiao
paper_title: VideoChat: Chat-Centric Video Understanding
paper_summary: In this study, we initiate an exploration into video understanding by
  introducing VideoChat, an end-to-end chat-centric video understanding system.
  It integrates video foundation models and large language models via a learnable
  neural interface, excelling in spatiotemporal reasoning, event localization,
  and causal relationship inference. To instructively tune this system, we
  propose a video-centric instruction dataset, composed of thousands of videos
  matched with detailed descriptions and conversations. This dataset emphasizes
  spatiotemporal reasoning and causal relationships, providing a valuable asset
  for training chat-centric video understanding systems. Preliminary qualitative
  experiments reveal our system's potential across a broad spectrum of video
  applications and set the standard for future research. Access our code and data
  at https://github.com/OpenGVLab/Ask-Anything
paper_ai_keywords: VideoChat end-to-end chat-centric video understanding system video foundation models large language models learnable neural interface spatiotemporal reasoning event localization causal relationship inference video-centric instruction dataset detailed descriptions conversations causality relationships qualitative experiments