OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video
  Generation
Text-to-video (T2V) generation has recently garnered significant attention
thanks to the large multi-modality model Sora. However, T2V generation still
faces two important challenges: 1) Lacking a precise open sourced high-quality
dataset. The previous popular video datasets, e.g. WebVid-10M and Panda-70M,
are either with low quality or too large for most research institutions.
Therefore, it is challenging but crucial to collect a precise high-quality
text-video pairs for T2V generation. 2) Ignoring to fully utilize textual
information. Recent T2V methods have focused on vision transformers, using a
simple cross attention module for video generation, which falls short of
thoroughly extracting semantic information from text prompt. To address these
issues, we introduce OpenVid-1M, a precise high-quality dataset with expressive
captions. This open-scenario dataset contains over 1 million text-video pairs,
facilitating research on T2V generation. Furthermore, we curate 433K 1080p
videos from OpenVid-1M to create OpenVidHD-0.4M, advancing high-definition
video generation. Additionally, we propose a novel Multi-modal Video Diffusion
Transformer (MVDiT) capable of mining both structure information from visual
tokens and semantic information from text tokens. Extensive experiments and
ablation studies verify the superiority of OpenVid-1M over previous datasets
and the effectiveness of our MVDiT.
668520f158b83aeb6bcd8415
Kepan Nan
claimed_verified
2024-07-10T14:25:29.898Z
64fafab460017eeec978a2a1
/avatars/c156bda4e3e48b449a35f38a1f25c497.svg
nkp
user
nkp37
668520f158b83aeb6bcd8416
Rui Xie
claimed_verified
2024-07-03T13:44:40.826Z
660a7ecf14cfe4973e0acfe1
/avatars/e488058397f2b7a617515a4f721a9a00.svg
Rui Xie
user
SherryX
668520f158b83aeb6bcd8417
Penghao Zhou
claimed_verified
2024-07-04T08:04:01.993Z
667e5d480f66814a82e56420
/avatars/33fd797c7690959fed7449666a2502bf.svg
Zhou
user
LeonZhouph
668520f158b83aeb6bcd8418
Tiehan Fan
claimed_verified
2024-08-16T08:20:03.519Z
65927f3b754092f6b1e187a7
https://cdn-avatars.huggingface.co/v1/production/uploads/65927f3b754092f6b1e187a7/gUrNvIQHmsl1vLwSUxpmL.jpeg
tiehan fan
user
AnonMegumi
668520f158b83aeb6bcd8419
Zhenheng Yang
claimed_verified
2024-07-04T08:04:03.843Z
6421183b69a2c2933882d652
/avatars/66813a8fa22915087cccd4dbfb945ca7.svg
Zhenheng Yang
user
zhenheny
668520f158b83aeb6bcd841a
Zhijie Chen
admin_assigned
2024-07-03T14:44:52.848Z
65d5a94036155e69ea717751
/avatars/9ff160b0a76dd56f913948d34dd045c8.svg
Zhijie Chen
user
Zhijie-Chen
668520f158b83aeb6bcd841b
Xiang Li
668520f158b83aeb6bcd841c
Jian Yang
668520f158b83aeb6bcd841d
Ying Tai
claimed_verified
2024-07-03T13:44:42.564Z
65734004769f3ee9bde1af10
/avatars/d6310ed861972fd691687d8f47413f33.svg
Ying Tai
user
yingtai
OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video
  Generation
Text-to-video (T2V) generation has recently garnered significant attention
thanks to the large multi-modality model Sora. However, T2V generation still
faces two important challenges: 1) Lacking a precise open sourced high-quality
dataset. The previous popular video datasets, e.g. WebVid-10M and Panda-70M,
are either with low quality or too large for most research institutions.
Therefore, it is challenging but crucial to collect a precise high-quality
text-video pairs for T2V generation. 2) Ignoring to fully utilize textual
information. Recent T2V methods have focused on vision transformers, using a
simple cross attention module for video generation, which falls short of
thoroughly extracting semantic information from text prompt. To address these
issues, we introduce OpenVid-1M, a precise high-quality dataset with expressive
captions. This open-scenario dataset contains over 1 million text-video pairs,
facilitating research on T2V generation. Furthermore, we curate 433K 1080p
videos from OpenVid-1M to create OpenVidHD-0.4M, advancing high-definition
video generation. Additionally, we propose a novel Multi-modal Video Diffusion
Transformer (MVDiT) capable of mining both structure information from visual
tokens and semantic information from text tokens. Extensive experiments and
ablation studies verify the superiority of OpenVid-1M over previous datasets
and the effectiveness of our MVDiT.
The paper introduces OpenVid-1M and OpenVidHD-0.4M datasets, along with MVDiT, a Multi-modal Video Diffusion Transformer, to enhance text-to-video generation by leveraging high-quality data and comprehensive text information.
Text-to-video generation Sora WebVid-10M Panda-70M open-scenario dataset text-video pairs cross attention module semantic information Multi-modal Video Diffusion Transformer MVDiT visual tokens text tokens video diffusion high-definition video generation