title: SEINE: Short-to-Long Video Diffusion Model for Generative Transition and
    Prediction
summary: Recently video generation has achieved substantial progress with realistic
  results. Nevertheless, existing AI-generated videos are usually very short
  clips ("shot-level") depicting a single scene. To deliver a coherent long video
  ("story-level"), it is desirable to have creative transition and prediction
  effects across different clips. This paper presents a short-to-long video
  diffusion model, SEINE, that focuses on generative transition and prediction.
  The goal is to generate high-quality long videos with smooth and creative
  transitions between scenes and varying lengths of shot-level videos.
  Specifically, we propose a random-mask video diffusion model to automatically
  generate transitions based on textual descriptions. By providing the images of
  different scenes as inputs, combined with text-based control, our model
  generates transition videos that ensure coherence and visual quality.
  Furthermore, the model can be readily extended to various tasks such as
  image-to-video animation and autoregressive video prediction. To conduct a
  comprehensive evaluation of this new generative task, we propose three
  assessing criteria for smooth and creative transition: temporal consistency,
  semantic similarity, and video-text semantic alignment. Extensive experiments
  validate the effectiveness of our approach over existing methods for generative
  transition and prediction, enabling the creation of story-level long videos.
  Project page: https://vchitect.github.io/SEINE-project/ .
paper_authors._id: 6541ca5d5242a5242f4a9852
paper_authors.name: Xinyuan Chen
paper_authors._id: 6541ca5d5242a5242f4a9853
paper_authors.name: Yaohui Wang
paper_authors._id: 6541ca5d5242a5242f4a9854
paper_authors.name: Lingjun Zhang
paper_authors._id: 6541ca5d5242a5242f4a9855
paper_authors.name: Shaobin Zhuang
paper_authors._id: 6541ca5d5242a5242f4a9856
paper_authors.name: Xin Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T07:50:42.331Z
paper_authors._id: 64a51faf9ec0167a99a4d114
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a51faf9ec0167a99a4d114/IlJS0mwkF_ZIebaQrbJIl.jpeg
paper_authors.fullname: Xin Ma
paper_authors.type: user
paper_authors.user: maxin-cn
paper_authors._id: 6541ca5d5242a5242f4a9857
paper_authors.name: Jiashuo Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-29T07:10:26.377Z
paper_authors._id: 64b89a14cf14c2fabe96664c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
paper_authors.fullname: Jiashuo Yu
paper_authors.type: user
paper_authors.user: awojustin
paper_authors._id: 6541ca5d5242a5242f4a9858
paper_authors.name: Yali Wang
paper_authors._id: 6541ca5d5242a5242f4a9859
paper_authors.name: Dahua Lin
paper_authors._id: 6541ca5d5242a5242f4a985a
paper_authors.name: Yu Qiao
paper_authors._id: 6541ca5d5242a5242f4a985b
paper_authors.name: Ziwei Liu
paper_title: SEINE: Short-to-Long Video Diffusion Model for Generative Transition and
    Prediction
paper_summary: Recently video generation has achieved substantial progress with realistic
  results. Nevertheless, existing AI-generated videos are usually very short
  clips ("shot-level") depicting a single scene. To deliver a coherent long video
  ("story-level"), it is desirable to have creative transition and prediction
  effects across different clips. This paper presents a short-to-long video
  diffusion model, SEINE, that focuses on generative transition and prediction.
  The goal is to generate high-quality long videos with smooth and creative
  transitions between scenes and varying lengths of shot-level videos.
  Specifically, we propose a random-mask video diffusion model to automatically
  generate transitions based on textual descriptions. By providing the images of
  different scenes as inputs, combined with text-based control, our model
  generates transition videos that ensure coherence and visual quality.
  Furthermore, the model can be readily extended to various tasks such as
  image-to-video animation and autoregressive video prediction. To conduct a
  comprehensive evaluation of this new generative task, we propose three
  assessing criteria for smooth and creative transition: temporal consistency,
  semantic similarity, and video-text semantic alignment. Extensive experiments
  validate the effectiveness of our approach over existing methods for generative
  transition and prediction, enabling the creation of story-level long videos.
  Project page: https://vchitect.github.io/SEINE-project/ .
paper_ai_keywords: video diffusion model SEINE gener open-source random-mask video diffusion model generative transition prediction textual descriptions temporal consistency semantic similarity video-text semantic alignment image-to-video animation autoregressive video prediction