STIV: Scalable Text and Image Conditioned Video Generation
The field of video generation has made remarkable advancements, yet there
remains a pressing need for a clear, systematic recipe that can guide the
development of robust and scalable models. In this work, we present a
comprehensive study that systematically explores the interplay of model
architectures, training recipes, and data curation strategies, culminating in a
simple and scalable text-image-conditioned video generation method, named STIV.
Our framework integrates image condition into a Diffusion Transformer (DiT)
through frame replacement, while incorporating text conditioning via a joint
image-text conditional classifier-free guidance. This design enables STIV to
perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks
simultaneously. Additionally, STIV can be easily extended to various
applications, such as video prediction, frame interpolation, multi-view
generation, and long video generation, etc. With comprehensive ablation studies
on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple
design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V,
surpassing both leading open and closed-source models like CogVideoX-5B, Pika,
Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result
of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and
extensible recipe for building cutting-edge video generation models, we aim to
empower future research and accelerate progress toward more versatile and
reliable video generation solutions.
67590ae29500a8d4bb970a6b
Zongyu Lin
claimed_verified
2025-02-05T10:15:43.449Z
6350304aa1e94729363e3ad6
/avatars/c142778f739c54f2de7780e311c29145.svg
lin
user
xiaoxiaolin
67590ae29500a8d4bb970a6c
Wei Liu
67590ae29500a8d4bb970a6d
Chen Chen
claimed_verified
2025-03-16T21:22:44.733Z
65409e53d07194a81ecdd427
/avatars/73a88ffe7f78f05b6fc8eb57e9f1e587.svg
Chen Chen
user
ultra7chen
67590ae29500a8d4bb970a6e
Jiasen Lu
admin_assigned
2024-12-11T10:14:42.640Z
62b6b0397523238923221df9
/avatars/77068771dd51df7519516cd502a88789.svg
Jiasenlu
user
Jiasenlu
67590ae29500a8d4bb970a6f
Wenze Hu
admin_assigned
2024-12-11T10:14:36.538Z
638807bd5c2df0349f443de9
/avatars/8593af4c3b716dfe17227618873dd3f2.svg
wenze
user
wenzehu
67590ae29500a8d4bb970a70
Tsu-Jui Fu
admin_assigned
2024-12-11T10:14:29.657Z
638cb2efc8912be69c149a46
https://cdn-avatars.huggingface.co/v1/production/uploads/638cb2efc8912be69c149a46/qxMNOWoSUG525DgNPpHcK.jpeg
Tsu-Jui Fu
user
tsujuifu
67590ae29500a8d4bb970a71
Jesse Allardice
admin_assigned
2024-12-11T10:14:23.262Z
6304992abad6ce7fc025a36a
/avatars/daf4c8a895336c7af45adff068e6f010.svg
Jesse Allardice
user
JesseAllardice
67590ae29500a8d4bb970a72
Zhengfeng Lai
claimed_verified
2024-12-11T09:19:53.394Z
66b5295f83425904fa7a1a6a
/avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
Zhengfeng Lai
user
jefflai
67590ae29500a8d4bb970a73
Liangchen Song
67590ae29500a8d4bb970a74
Bowen Zhang
67590ae29500a8d4bb970a75
Cha Chen
67590ae29500a8d4bb970a76
Yiran Fei
67590ae29500a8d4bb970a77
Yifan Jiang
67590ae29500a8d4bb970a78
Lezhi Li
67590ae29500a8d4bb970a79
Yizhou Sun
admin_assigned
2024-12-11T10:13:07.473Z
67091ab8a6ad85d9c8dade7f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/2Mr-EVQJaeMFydmhEXXBI.png
Yizhou Sun
user
HoSaiiRai
67590ae29500a8d4bb970a7a
Kai-Wei Chang
admin_assigned
2024-12-11T10:12:58.775Z
60b7b9d71b90c5d07c23fbd0
https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
Kai-Wei Chang
user
kaiweichang
67590ae29500a8d4bb970a7b
Yinfei Yang
admin_assigned
2024-12-11T10:12:52.506Z
64b762568c632fbca942a405
/avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
Yinfei Yang
user
yinfeiy
STIV: Scalable Text and Image Conditioned Video Generation
The field of video generation has made remarkable advancements, yet there
remains a pressing need for a clear, systematic recipe that can guide the
development of robust and scalable models. In this work, we present a
comprehensive study that systematically explores the interplay of model
architectures, training recipes, and data curation strategies, culminating in a
simple and scalable text-image-conditioned video generation method, named STIV.
Our framework integrates image condition into a Diffusion Transformer (DiT)
through frame replacement, while incorporating text conditioning via a joint
image-text conditional classifier-free guidance. This design enables STIV to
perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks
simultaneously. Additionally, STIV can be easily extended to various
applications, such as video prediction, frame interpolation, multi-view
generation, and long video generation, etc. With comprehensive ablation studies
on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple
design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V,
surpassing both leading open and closed-source models like CogVideoX-5B, Pika,
Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result
of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and
extensible recipe for building cutting-edge video generation models, we aim to
empower future research and accelerate progress toward more versatile and
reliable video generation solutions.
STIV, a text-image-conditioned video generation method integrating Diffusion Transformer and classifier-free guidance, achieves state-of-the-art performance in text-to-video, text-image-to-video, and image-to-video tasks.
video generation model architectures training recipes data curation strategies Diffusion Transformer DiT frame replacement joint image-text conditional classifier-free guidance text-to-video text-image-to-video video prediction frame interpolation multi-view generation long video generation VBench CogVideoX-5B Pika Kling Gen-3 image-to-video