title: I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion
    Models
summary: Video synthesis has recently made remarkable strides benefiting from the
  rapid development of diffusion models. However, it still encounters challenges
  in terms of semantic accuracy, clarity and spatio-temporal continuity. They
  primarily arise from the scarcity of well-aligned text-video data and the
  complex inherent structure of videos, making it difficult for the model to
  simultaneously ensure semantic and qualitative excellence. In this report, we
  propose a cascaded I2VGen-XL approach that enhances model performance by
  decoupling these two factors and ensures the alignment of the input data by
  utilizing static images as a form of crucial guidance. I2VGen-XL consists of
  two stages: i) the base stage guarantees coherent semantics and preserves
  content from input images by using two hierarchical encoders, and ii) the
  refinement stage enhances the video's details by incorporating an additional
  brief text and improves the resolution to 1280times720. To improve the
  diversity, we collect around 35 million single-shot text-video pairs and 6
  billion text-image pairs to optimize the model. By this means, I2VGen-XL can
  simultaneously enhance the semantic accuracy, continuity of details and clarity
  of generated videos. Through extensive experiments, we have investigated the
  underlying principles of I2VGen-XL and compared it with current top methods,
  which can demonstrate its effectiveness on diverse data. The source code and
  models will be publicly available at https://i2vgen-xl.github.io.
paper_authors._id: 654af0fce4432d6c7b0dfe91
paper_authors.name: Shiwei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-08T08:27:25.768Z
paper_authors._id: 63a1200045edac9f7508bae9
paper_authors.avatarUrl: /avatars/e84f0b045f32c5b8b4da43458650b925.svg
paper_authors.fullname: Shiwei Zhang
paper_authors.type: user
paper_authors.user: StevenZhang
paper_authors._id: 654af0fce4432d6c7b0dfe92
paper_authors.name: Jiayu Wang
paper_authors._id: 654af0fce4432d6c7b0dfe93
paper_authors.name: Yingya Zhang
paper_authors._id: 654af0fce4432d6c7b0dfe94
paper_authors.name: Kang Zhao
paper_authors._id: 654af0fce4432d6c7b0dfe95
paper_authors.name: Hangjie Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-08T08:37:26.728Z
paper_authors._id: 649d54b314afbb10ce2a9eeb
paper_authors.avatarUrl: /avatars/15c325d8c2273ff63569f23015e98486.svg
paper_authors.fullname: Hangjie Yuan
paper_authors.type: user
paper_authors.user: JacobYuan
paper_authors._id: 654af0fce4432d6c7b0dfe96
paper_authors.name: Zhiwu Qin
paper_authors._id: 654af0fce4432d6c7b0dfe97
paper_authors.name: Xiang Wang
paper_authors._id: 654af0fce4432d6c7b0dfe98
paper_authors.name: Deli Zhao
paper_authors._id: 654af0fce4432d6c7b0dfe99
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-08T08:38:48.065Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion
    Models
paper_summary: Video synthesis has recently made remarkable strides benefiting from the
  rapid development of diffusion models. However, it still encounters challenges
  in terms of semantic accuracy, clarity and spatio-temporal continuity. They
  primarily arise from the scarcity of well-aligned text-video data and the
  complex inherent structure of videos, making it difficult for the model to
  simultaneously ensure semantic and qualitative excellence. In this report, we
  propose a cascaded I2VGen-XL approach that enhances model performance by
  decoupling these two factors and ensures the alignment of the input data by
  utilizing static images as a form of crucial guidance. I2VGen-XL consists of
  two stages: i) the base stage guarantees coherent semantics and preserves
  content from input images by using two hierarchical encoders, and ii) the
  refinement stage enhances the video's details by incorporating an additional
  brief text and improves the resolution to 1280times720. To improve the
  diversity, we collect around 35 million single-shot text-video pairs and 6
  billion text-image pairs to optimize the model. By this means, I2VGen-XL can
  simultaneously enhance the semantic accuracy, continuity of details and clarity
  of generated videos. Through extensive experiments, we have investigated the
  underlying principles of I2VGen-XL and compared it with current top methods,
  which can demonstrate its effectiveness on diverse data. The source code and
  models will be publicly available at https://i2vgen-xl.github.io.
paper_ai_keywords: cascaded I2VGen-XL diffusion models semantic accuracy spatio-temporal continuity text-video data hierarchical encoders refinement stage text and image pairs video's details