LVD-2M: A Long-take Video Dataset with Temporally Dense Captions
The efficacy of video generation models heavily depends on the quality of
their training datasets. Most previous video generation models are trained on
short video clips, while recently there has been increasing interest in
training long video generation models directly on longer videos. However, the
lack of such high-quality long videos impedes the advancement of long video
generation. To promote research in long video generation, we desire a new
dataset with four key features essential for training long video generation
models: (1) long videos covering at least 10 seconds, (2) long-take videos
without cuts, (3) large motion and diverse contents, and (4) temporally dense
captions. To achieve this, we introduce a new pipeline for selecting
high-quality long-take videos and generating temporally dense captions.
Specifically, we define a set of metrics to quantitatively assess video quality
including scene cuts, dynamic degrees, and semantic-level quality, enabling us
to filter high-quality long-take videos from a large amount of source videos.
Subsequently, we develop a hierarchical video captioning pipeline to annotate
long videos with temporally-dense captions. With this pipeline, we curate the
first long-take video dataset, LVD-2M, comprising 2 million long-take videos,
each covering more than 10 seconds and annotated with temporally dense
captions. We further validate the effectiveness of LVD-2M by fine-tuning video
generation models to generate long videos with dynamic motions. We believe our
work will significantly contribute to future research in long video generation.
670f3501596612949a2dbff0
Tianwei Xiong
claimed_verified
2024-10-16T09:43:36.713Z
668125557b50b433cda2a211
https://cdn-avatars.huggingface.co/v1/production/uploads/668125557b50b433cda2a211/j3z3wT5Rv9IyUKtbzQpnc.png
Tianwei Xiong
user
YuuTennYi
670f3501596612949a2dbff1
Yuqing Wang
claimed_verified
2024-10-17T11:26:48.804Z
63ea23b9dedfeebe54d02bdf
/avatars/4d9f9a546aa8c63e277161ea700075c4.svg
Yuqing Wang
user
Epiphqny
670f3501596612949a2dbff2
Daquan Zhou
670f3501596612949a2dbff3
Zhijie Lin
670f3501596612949a2dbff4
Jiashi Feng
670f3501596612949a2dbff5
Xihui Liu
claimed_verified
2024-10-16T09:43:38.536Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
LVD-2M: A Long-take Video Dataset with Temporally Dense Captions
The efficacy of video generation models heavily depends on the quality of
their training datasets. Most previous video generation models are trained on
short video clips, while recently there has been increasing interest in
training long video generation models directly on longer videos. However, the
lack of such high-quality long videos impedes the advancement of long video
generation. To promote research in long video generation, we desire a new
dataset with four key features essential for training long video generation
models: (1) long videos covering at least 10 seconds, (2) long-take videos
without cuts, (3) large motion and diverse contents, and (4) temporally dense
captions. To achieve this, we introduce a new pipeline for selecting
high-quality long-take videos and generating temporally dense captions.
Specifically, we define a set of metrics to quantitatively assess video quality
including scene cuts, dynamic degrees, and semantic-level quality, enabling us
to filter high-quality long-take videos from a large amount of source videos.
Subsequently, we develop a hierarchical video captioning pipeline to annotate
long videos with temporally-dense captions. With this pipeline, we curate the
first long-take video dataset, LVD-2M, comprising 2 million long-take videos,
each covering more than 10 seconds and annotated with temporally dense
captions. We further validate the effectiveness of LVD-2M by fine-tuning video
generation models to generate long videos with dynamic motions. We believe our
work will significantly contribute to future research in long video generation.
A new dataset, LVD-2M, is curated to enhance long video generation by providing high-quality, long-take videos with temporally dense captions.
video generation models long video generation high-quality long videos long-take videos scene cuts dynamic degrees semantic-level quality hierarchical video captioning pipeline temporally dense captions