MiraData: A Large-Scale Video Dataset with Long Durations and Structured
  Captions
Sora's high-motion intensity and long consistent videos have significantly
impacted the field of video generation, attracting unprecedented attention.
However, existing publicly available datasets are inadequate for generating
Sora-like videos, as they mainly contain short videos with low motion intensity
and brief captions. To address these issues, we propose MiraData, a
high-quality video dataset that surpasses previous ones in video duration,
caption detail, motion strength, and visual quality. We curate MiraData from
diverse, manually selected sources and meticulously process the data to obtain
semantically consistent clips. GPT-4V is employed to annotate structured
captions, providing detailed descriptions from four different perspectives
along with a summarized dense caption. To better assess temporal consistency
and motion intensity in video generation, we introduce MiraBench, which
enhances existing benchmarks by adding 3D consistency and tracking-based motion
strength metrics. MiraBench includes 150 evaluation prompts and 17 metrics
covering temporal consistency, motion strength, 3D consistency, visual quality,
text-video alignment, and distribution similarity. To demonstrate the utility
and effectiveness of MiraData, we conduct experiments using our DiT-based video
generation model, MiraDiT. The experimental results on MiraBench demonstrate
the superiority of MiraData, especially in motion strength.
668df5fe3d0bc7c475c56e07
Xuan Ju
claimed_verified
2024-07-30T07:26:23.310Z
62d4577bc85b0fcf7fde39bb
/avatars/a3a5729e33ae89ce9ba408830db3c835.svg
Xuan Ju
user
juxuan27
668df5fe3d0bc7c475c56e08
Yiming Gao
668df5fe3d0bc7c475c56e09
Zhaoyang Zhang
claimed_verified
2024-07-10T08:19:34.562Z
658409ceca19ccf6d9989add
/avatars/3ac1dbd25e52435185babdeb3da28875.svg
Zhaoyang Zhang
user
ZyZcuhk
668df5fe3d0bc7c475c56e0a
Ziyang Yuan
668df5fe3d0bc7c475c56e0b
Xintao Wang
admin_assigned
2024-07-10T10:42:14.779Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
668df5fe3d0bc7c475c56e0c
Ailing Zeng
668df5fe3d0bc7c475c56e0d
Yu Xiong
668df5fe3d0bc7c475c56e0e
Qiang Xu
668df5fe3d0bc7c475c56e0f
Ying Shan
MiraData: A Large-Scale Video Dataset with Long Durations and Structured
  Captions
Sora's high-motion intensity and long consistent videos have significantly
impacted the field of video generation, attracting unprecedented attention.
However, existing publicly available datasets are inadequate for generating
Sora-like videos, as they mainly contain short videos with low motion intensity
and brief captions. To address these issues, we propose MiraData, a
high-quality video dataset that surpasses previous ones in video duration,
caption detail, motion strength, and visual quality. We curate MiraData from
diverse, manually selected sources and meticulously process the data to obtain
semantically consistent clips. GPT-4V is employed to annotate structured
captions, providing detailed descriptions from four different perspectives
along with a summarized dense caption. To better assess temporal consistency
and motion intensity in video generation, we introduce MiraBench, which
enhances existing benchmarks by adding 3D consistency and tracking-based motion
strength metrics. MiraBench includes 150 evaluation prompts and 17 metrics
covering temporal consistency, motion strength, 3D consistency, visual quality,
text-video alignment, and distribution similarity. To demonstrate the utility
and effectiveness of MiraData, we conduct experiments using our DiT-based video
generation model, MiraDiT. The experimental results on MiraBench demonstrate
the superiority of MiraData, especially in motion strength.
A new video dataset, MiraData, and benchmark, MiraBench, are introduced to improve video generation quality, particularly in motion intensity and temporal consistency, using GPT-4V annotations and DiT-based models.
MiraData MiraBench GPT-4V DiT-based video generation model 3D consistency tracking-based motion strength metrics