title: MiraData: A Large-Scale Video Dataset with Long Durations and Structured
    Captions
summary: Sora's high-motion intensity and long consistent videos have significantly
  impacted the field of video generation, attracting unprecedented attention.
  However, existing publicly available datasets are inadequate for generating
  Sora-like videos, as they mainly contain short videos with low motion intensity
  and brief captions. To address these issues, we propose MiraData, a
  high-quality video dataset that surpasses previous ones in video duration,
  caption detail, motion strength, and visual quality. We curate MiraData from
  diverse, manually selected sources and meticulously process the data to obtain
  semantically consistent clips. GPT-4V is employed to annotate structured
  captions, providing detailed descriptions from four different perspectives
  along with a summarized dense caption. To better assess temporal consistency
  and motion intensity in video generation, we introduce MiraBench, which
  enhances existing benchmarks by adding 3D consistency and tracking-based motion
  strength metrics. MiraBench includes 150 evaluation prompts and 17 metrics
  covering temporal consistency, motion strength, 3D consistency, visual quality,
  text-video alignment, and distribution similarity. To demonstrate the utility
  and effectiveness of MiraData, we conduct experiments using our DiT-based video
  generation model, MiraDiT. The experimental results on MiraBench demonstrate
  the superiority of MiraData, especially in motion strength.
paper_authors._id: 668df5fe3d0bc7c475c56e07
paper_authors.name: Xuan Ju
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T07:26:23.310Z
paper_authors._id: 62d4577bc85b0fcf7fde39bb
paper_authors.avatarUrl: /avatars/a3a5729e33ae89ce9ba408830db3c835.svg
paper_authors.fullname: Xuan Ju
paper_authors.type: user
paper_authors.user: juxuan27
paper_authors._id: 668df5fe3d0bc7c475c56e08
paper_authors.name: Yiming Gao
paper_authors._id: 668df5fe3d0bc7c475c56e09
paper_authors.name: Zhaoyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T08:19:34.562Z
paper_authors._id: 658409ceca19ccf6d9989add
paper_authors.avatarUrl: /avatars/3ac1dbd25e52435185babdeb3da28875.svg
paper_authors.fullname: Zhaoyang Zhang
paper_authors.type: user
paper_authors.user: ZyZcuhk
paper_authors._id: 668df5fe3d0bc7c475c56e0a
paper_authors.name: Ziyang Yuan
paper_authors._id: 668df5fe3d0bc7c475c56e0b
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T10:42:14.779Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 668df5fe3d0bc7c475c56e0c
paper_authors.name: Ailing Zeng
paper_authors._id: 668df5fe3d0bc7c475c56e0d
paper_authors.name: Yu Xiong
paper_authors._id: 668df5fe3d0bc7c475c56e0e
paper_authors.name: Qiang Xu
paper_authors._id: 668df5fe3d0bc7c475c56e0f
paper_authors.name: Ying Shan
paper_title: MiraData: A Large-Scale Video Dataset with Long Durations and Structured
    Captions
paper_summary: Sora's high-motion intensity and long consistent videos have significantly
  impacted the field of video generation, attracting unprecedented attention.
  However, existing publicly available datasets are inadequate for generating
  Sora-like videos, as they mainly contain short videos with low motion intensity
  and brief captions. To address these issues, we propose MiraData, a
  high-quality video dataset that surpasses previous ones in video duration,
  caption detail, motion strength, and visual quality. We curate MiraData from
  diverse, manually selected sources and meticulously process the data to obtain
  semantically consistent clips. GPT-4V is employed to annotate structured
  captions, providing detailed descriptions from four different perspectives
  along with a summarized dense caption. To better assess temporal consistency
  and motion intensity in video generation, we introduce MiraBench, which
  enhances existing benchmarks by adding 3D consistency and tracking-based motion
  strength metrics. MiraBench includes 150 evaluation prompts and 17 metrics
  covering temporal consistency, motion strength, 3D consistency, visual quality,
  text-video alignment, and distribution similarity. To demonstrate the utility
  and effectiveness of MiraData, we conduct experiments using our DiT-based video
  generation model, MiraDiT. The experimental results on MiraBench demonstrate
  the superiority of MiraData, especially in motion strength.
paper_ai_summary: A new video dataset, MiraData, and benchmark, MiraBench, are introduced to improve video generation quality, particularly in motion intensity and temporal consistency, using GPT-4V annotations and DiT-based models.
paper_ai_keywords: MiraData MiraBench GPT-4V DiT-based video generation model 3D consistency tracking-based motion strength metrics