VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video
  Generation
Text-to-video generative models convert textual prompts into dynamic visual
content, offering wide-ranging applications in film production, gaming, and
education. However, their real-world performance often falls short of user
expectations. One key reason is that these models have not been trained on
videos related to some topics users want to create. In this paper, we propose
VideoUFO, the first Video dataset specifically curated to align with Users'
FOcus in real-world scenarios. Beyond this, our VideoUFO also features: (1)
minimal (0.29%) overlap with existing video datasets, and (2) videos
searched exclusively via YouTube's official API under the Creative Commons
license. These two attributes provide future researchers with greater freedom
to broaden their training sources. The VideoUFO comprises over 1.09 million
video clips, each paired with both a brief and a detailed caption
(description). Specifically, through clustering, we first identify 1,291
user-focused topics from the million-scale real text-to-video prompt dataset,
VidProM. Then, we use these topics to retrieve videos from YouTube, split the
retrieved videos into clips, and generate both brief and detailed captions for
each clip. After verifying the clips with specified topics, we are left with
about 1.09 million video clips. Our experiments reveal that (1) current 16
text-to-video models do not achieve consistent performance across all
user-focused topics; and (2) a simple model trained on VideoUFO outperforms
others on worst-performing topics. The dataset is publicly available at
https://huggingface.co/datasets/WenhaoWang/VideoUFO under the CC BY 4.0
License.
67c68f7828a037872c5ce5bb
Wenhao Wang
admin_assigned
2025-03-04T11:14:37.907Z
62b32a4429a410b7f6b06710
https://cdn-avatars.huggingface.co/v1/production/uploads/62b32a4429a410b7f6b06710/VzgvmnlYZWuifZTkIkCxy.jpeg
Wenhao Wang
user
WenhaoWang
67c68f7828a037872c5ce5bc
Yi Yang
VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video
  Generation
Text-to-video generative models convert textual prompts into dynamic visual
content, offering wide-ranging applications in film production, gaming, and
education. However, their real-world performance often falls short of user
expectations. One key reason is that these models have not been trained on
videos related to some topics users want to create. In this paper, we propose
VideoUFO, the first Video dataset specifically curated to align with Users'
FOcus in real-world scenarios. Beyond this, our VideoUFO also features: (1)
minimal (0.29%) overlap with existing video datasets, and (2) videos
searched exclusively via YouTube's official API under the Creative Commons
license. These two attributes provide future researchers with greater freedom
to broaden their training sources. The VideoUFO comprises over 1.09 million
video clips, each paired with both a brief and a detailed caption
(description). Specifically, through clustering, we first identify 1,291
user-focused topics from the million-scale real text-to-video prompt dataset,
VidProM. Then, we use these topics to retrieve videos from YouTube, split the
retrieved videos into clips, and generate both brief and detailed captions for
each clip. After verifying the clips with specified topics, we are left with
about 1.09 million video clips. Our experiments reveal that (1) current 16
text-to-video models do not achieve consistent performance across all
user-focused topics; and (2) a simple model trained on VideoUFO outperforms
others on worst-performing topics. The dataset is publicly available at
https://huggingface.co/datasets/WenhaoWang/VideoUFO under the CC BY 4.0
License.
A new dataset, VideoUFO, addresses real-world limitations in text-to-video generative models by focusing on user-centric topics, enhancing training sources and model performance on poorly-covered subjects.
text-to-video generative models VideoUFO user-focused topics video datasets clustering YouTube's official API Creative Commons license text-to-video prompt dataset VidProM video clips captions model performance