title: STIV: Scalable Text and Image Conditioned Video Generation
summary: The field of video generation has made remarkable advancements, yet there
  remains a pressing need for a clear, systematic recipe that can guide the
  development of robust and scalable models. In this work, we present a
  comprehensive study that systematically explores the interplay of model
  architectures, training recipes, and data curation strategies, culminating in a
  simple and scalable text-image-conditioned video generation method, named STIV.
  Our framework integrates image condition into a Diffusion Transformer (DiT)
  through frame replacement, while incorporating text conditioning via a joint
  image-text conditional classifier-free guidance. This design enables STIV to
  perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks
  simultaneously. Additionally, STIV can be easily extended to various
  applications, such as video prediction, frame interpolation, multi-view
  generation, and long video generation, etc. With comprehensive ablation studies
  on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple
  design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V,
  surpassing both leading open and closed-source models like CogVideoX-5B, Pika,
  Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result
  of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and
  extensible recipe for building cutting-edge video generation models, we aim to
  empower future research and accelerate progress toward more versatile and
  reliable video generation solutions.
paper_authors._id: 67590ae29500a8d4bb970a6b
paper_authors.name: Zongyu Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T10:15:43.449Z
paper_authors._id: 6350304aa1e94729363e3ad6
paper_authors.avatarUrl: /avatars/c142778f739c54f2de7780e311c29145.svg
paper_authors.fullname: lin
paper_authors.type: user
paper_authors.user: xiaoxiaolin
paper_authors._id: 67590ae29500a8d4bb970a6c
paper_authors.name: Wei Liu
paper_authors._id: 67590ae29500a8d4bb970a6d
paper_authors.name: Chen Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:22:44.733Z
paper_authors._id: 65409e53d07194a81ecdd427
paper_authors.avatarUrl: /avatars/73a88ffe7f78f05b6fc8eb57e9f1e587.svg
paper_authors.fullname: Chen Chen
paper_authors.type: user
paper_authors.user: ultra7chen
paper_authors._id: 67590ae29500a8d4bb970a6e
paper_authors.name: Jiasen Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:14:42.640Z
paper_authors._id: 62b6b0397523238923221df9
paper_authors.avatarUrl: /avatars/77068771dd51df7519516cd502a88789.svg
paper_authors.fullname: Jiasenlu
paper_authors.type: user
paper_authors.user: Jiasenlu
paper_authors._id: 67590ae29500a8d4bb970a6f
paper_authors.name: Wenze Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:14:36.538Z
paper_authors._id: 638807bd5c2df0349f443de9
paper_authors.avatarUrl: /avatars/8593af4c3b716dfe17227618873dd3f2.svg
paper_authors.fullname: wenze
paper_authors.type: user
paper_authors.user: wenzehu
paper_authors._id: 67590ae29500a8d4bb970a70
paper_authors.name: Tsu-Jui Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:14:29.657Z
paper_authors._id: 638cb2efc8912be69c149a46
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638cb2efc8912be69c149a46/qxMNOWoSUG525DgNPpHcK.jpeg
paper_authors.fullname: Tsu-Jui Fu
paper_authors.type: user
paper_authors.user: tsujuifu
paper_authors._id: 67590ae29500a8d4bb970a71
paper_authors.name: Jesse Allardice
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:14:23.262Z
paper_authors._id: 6304992abad6ce7fc025a36a
paper_authors.avatarUrl: /avatars/daf4c8a895336c7af45adff068e6f010.svg
paper_authors.fullname: Jesse Allardice
paper_authors.type: user
paper_authors.user: JesseAllardice
paper_authors._id: 67590ae29500a8d4bb970a72
paper_authors.name: Zhengfeng Lai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:19:53.394Z
paper_authors._id: 66b5295f83425904fa7a1a6a
paper_authors.avatarUrl: /avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
paper_authors.fullname: Zhengfeng Lai
paper_authors.type: user
paper_authors.user: jefflai
paper_authors._id: 67590ae29500a8d4bb970a73
paper_authors.name: Liangchen Song
paper_authors._id: 67590ae29500a8d4bb970a74
paper_authors.name: Bowen Zhang
paper_authors._id: 67590ae29500a8d4bb970a75
paper_authors.name: Cha Chen
paper_authors._id: 67590ae29500a8d4bb970a76
paper_authors.name: Yiran Fei
paper_authors._id: 67590ae29500a8d4bb970a77
paper_authors.name: Yifan Jiang
paper_authors._id: 67590ae29500a8d4bb970a78
paper_authors.name: Lezhi Li
paper_authors._id: 67590ae29500a8d4bb970a79
paper_authors.name: Yizhou Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:13:07.473Z
paper_authors._id: 67091ab8a6ad85d9c8dade7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/2Mr-EVQJaeMFydmhEXXBI.png
paper_authors.fullname: Yizhou Sun
paper_authors.type: user
paper_authors.user: HoSaiiRai
paper_authors._id: 67590ae29500a8d4bb970a7a
paper_authors.name: Kai-Wei Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:12:58.775Z
paper_authors._id: 60b7b9d71b90c5d07c23fbd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
paper_authors.fullname: Kai-Wei Chang
paper_authors.type: user
paper_authors.user: kaiweichang
paper_authors._id: 67590ae29500a8d4bb970a7b
paper_authors.name: Yinfei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:12:52.506Z
paper_authors._id: 64b762568c632fbca942a405
paper_authors.avatarUrl: /avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
paper_authors.fullname: Yinfei Yang
paper_authors.type: user
paper_authors.user: yinfeiy
paper_title: STIV: Scalable Text and Image Conditioned Video Generation
paper_summary: The field of video generation has made remarkable advancements, yet there
  remains a pressing need for a clear, systematic recipe that can guide the
  development of robust and scalable models. In this work, we present a
  comprehensive study that systematically explores the interplay of model
  architectures, training recipes, and data curation strategies, culminating in a
  simple and scalable text-image-conditioned video generation method, named STIV.
  Our framework integrates image condition into a Diffusion Transformer (DiT)
  through frame replacement, while incorporating text conditioning via a joint
  image-text conditional classifier-free guidance. This design enables STIV to
  perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks
  simultaneously. Additionally, STIV can be easily extended to various
  applications, such as video prediction, frame interpolation, multi-view
  generation, and long video generation, etc. With comprehensive ablation studies
  on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple
  design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V,
  surpassing both leading open and closed-source models like CogVideoX-5B, Pika,
  Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result
  of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and
  extensible recipe for building cutting-edge video generation models, we aim to
  empower future research and accelerate progress toward more versatile and
  reliable video generation solutions.
paper_ai_keywords: Diffusion Transformer (DiT) frame replacement joint image-text conditional classifier-free guidance text-to-video (T2V) text-image-to-video (TI2V) video prediction frame interpolation multi-view generation long video generation VBench T2V VBench I2V