title: OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video
    Generation
summary: Text-to-video (T2V) generation has recently garnered significant attention
  thanks to the large multi-modality model Sora. However, T2V generation still
  faces two important challenges: 1) Lacking a precise open sourced high-quality
  dataset. The previous popular video datasets, e.g. WebVid-10M and Panda-70M,
  are either with low quality or too large for most research institutions.
  Therefore, it is challenging but crucial to collect a precise high-quality
  text-video pairs for T2V generation. 2) Ignoring to fully utilize textual
  information. Recent T2V methods have focused on vision transformers, using a
  simple cross attention module for video generation, which falls short of
  thoroughly extracting semantic information from text prompt. To address these
  issues, we introduce OpenVid-1M, a precise high-quality dataset with expressive
  captions. This open-scenario dataset contains over 1 million text-video pairs,
  facilitating research on T2V generation. Furthermore, we curate 433K 1080p
  videos from OpenVid-1M to create OpenVidHD-0.4M, advancing high-definition
  video generation. Additionally, we propose a novel Multi-modal Video Diffusion
  Transformer (MVDiT) capable of mining both structure information from visual
  tokens and semantic information from text tokens. Extensive experiments and
  ablation studies verify the superiority of OpenVid-1M over previous datasets
  and the effectiveness of our MVDiT.
paper_authors._id: 668520f158b83aeb6bcd8415
paper_authors.name: Kepan Nan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T14:25:29.898Z
paper_authors._id: 64fafab460017eeec978a2a1
paper_authors.avatarUrl: /avatars/c156bda4e3e48b449a35f38a1f25c497.svg
paper_authors.fullname: nkp
paper_authors.type: user
paper_authors.user: nkp37
paper_authors._id: 668520f158b83aeb6bcd8416
paper_authors.name: Rui Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T13:44:40.826Z
paper_authors._id: 660a7ecf14cfe4973e0acfe1
paper_authors.avatarUrl: /avatars/e488058397f2b7a617515a4f721a9a00.svg
paper_authors.fullname: Rui Xie
paper_authors.type: user
paper_authors.user: SherryX
paper_authors._id: 668520f158b83aeb6bcd8417
paper_authors.name: Penghao Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-04T08:04:01.993Z
paper_authors._id: 667e5d480f66814a82e56420
paper_authors.avatarUrl: /avatars/33fd797c7690959fed7449666a2502bf.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: LeonZhouph
paper_authors._id: 668520f158b83aeb6bcd8418
paper_authors.name: Tiehan Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-16T08:20:03.519Z
paper_authors._id: 65927f3b754092f6b1e187a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65927f3b754092f6b1e187a7/gUrNvIQHmsl1vLwSUxpmL.jpeg
paper_authors.fullname: tiehan fan
paper_authors.type: user
paper_authors.user: AnonMegumi
paper_authors._id: 668520f158b83aeb6bcd8419
paper_authors.name: Zhenheng Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-04T08:04:03.843Z
paper_authors._id: 6421183b69a2c2933882d652
paper_authors.avatarUrl: /avatars/66813a8fa22915087cccd4dbfb945ca7.svg
paper_authors.fullname: Zhenheng Yang
paper_authors.type: user
paper_authors.user: zhenheny
paper_authors._id: 668520f158b83aeb6bcd841a
paper_authors.name: Zhijie Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T14:44:52.848Z
paper_authors._id: 65d5a94036155e69ea717751
paper_authors.avatarUrl: /avatars/9ff160b0a76dd56f913948d34dd045c8.svg
paper_authors.fullname: Zhijie Chen
paper_authors.type: user
paper_authors.user: Zhijie-Chen
paper_authors._id: 668520f158b83aeb6bcd841b
paper_authors.name: Xiang Li
paper_authors._id: 668520f158b83aeb6bcd841c
paper_authors.name: Jian Yang
paper_authors._id: 668520f158b83aeb6bcd841d
paper_authors.name: Ying Tai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T13:44:42.564Z
paper_authors._id: 65734004769f3ee9bde1af10
paper_authors.avatarUrl: /avatars/d6310ed861972fd691687d8f47413f33.svg
paper_authors.fullname: Ying Tai
paper_authors.type: user
paper_authors.user: yingtai
paper_title: OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video
    Generation
paper_summary: Text-to-video (T2V) generation has recently garnered significant attention
  thanks to the large multi-modality model Sora. However, T2V generation still
  faces two important challenges: 1) Lacking a precise open sourced high-quality
  dataset. The previous popular video datasets, e.g. WebVid-10M and Panda-70M,
  are either with low quality or too large for most research institutions.
  Therefore, it is challenging but crucial to collect a precise high-quality
  text-video pairs for T2V generation. 2) Ignoring to fully utilize textual
  information. Recent T2V methods have focused on vision transformers, using a
  simple cross attention module for video generation, which falls short of
  thoroughly extracting semantic information from text prompt. To address these
  issues, we introduce OpenVid-1M, a precise high-quality dataset with expressive
  captions. This open-scenario dataset contains over 1 million text-video pairs,
  facilitating research on T2V generation. Furthermore, we curate 433K 1080p
  videos from OpenVid-1M to create OpenVidHD-0.4M, advancing high-definition
  video generation. Additionally, we propose a novel Multi-modal Video Diffusion
  Transformer (MVDiT) capable of mining both structure information from visual
  tokens and semantic information from text tokens. Extensive experiments and
  ablation studies verify the superiority of OpenVid-1M over previous datasets
  and the effectiveness of our MVDiT.
paper_ai_keywords: Multi-modal Video Diffusion Transformer MVDiT vision transformers cross attention text-video pairs