title: FreeInit: Bridging Initialization Gap in Video Diffusion Models
summary: Though diffusion-based video generation has witnessed rapid progress, the
  inference results of existing models still exhibit unsatisfactory temporal
  consistency and unnatural dynamics. In this paper, we delve deep into the noise
  initialization of video diffusion models, and discover an implicit
  training-inference gap that attributes to the unsatisfactory inference quality.
  Our key findings are: 1) the spatial-temporal frequency distribution of the
  initial latent at inference is intrinsically different from that for training,
  and 2) the denoising process is significantly influenced by the low-frequency
  components of the initial noise. Motivated by these observations, we propose a
  concise yet effective inference sampling strategy, FreeInit, which
  significantly improves temporal consistency of videos generated by diffusion
  models. Through iteratively refining the spatial-temporal low-frequency
  components of the initial latent during inference, FreeInit is able to
  compensate the initialization gap between training and inference, thus
  effectively improving the subject appearance and temporal consistency of
  generation results. Extensive experiments demonstrate that FreeInit
  consistently enhances the generation results of various text-to-video
  generation models without additional training.
paper_authors._id: 657924094d989b0a68bf85f2
paper_authors.name: Tianxing Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T08:34:09.020Z
paper_authors._id: 63185a8973e9e4aa51dbc0ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676359176854-63185a8973e9e4aa51dbc0ce.jpeg
paper_authors.fullname: Tianxing Wu
paper_authors.type: user
paper_authors.user: TianxingWu
paper_authors._id: 657924094d989b0a68bf85f3
paper_authors.name: Chenyang Si
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T08:34:18.858Z
paper_authors._id: 635f8ed47c05eb9f59963d3a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
paper_authors.fullname: ChenyangSi
paper_authors.type: user
paper_authors.user: ChenyangSi
paper_authors._id: 657924094d989b0a68bf85f4
paper_authors.name: Yuming Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T08:33:46.802Z
paper_authors._id: 629c95b7a5d6f5fe10e6ed45
paper_authors.avatarUrl: /avatars/c9a5f0e7fae3a079d26db266e9ff90a3.svg
paper_authors.fullname: Yuming Jiang
paper_authors.type: user
paper_authors.user: yumingj
paper_authors._id: 657924094d989b0a68bf85f5
paper_authors.name: Ziqi Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-13T08:32:45.214Z
paper_authors._id: 60efe7fa0d920bc7805cada5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png
paper_authors.fullname: Ziqi Huang
paper_authors.type: user
paper_authors.user: Ziqi
paper_authors._id: 657924094d989b0a68bf85f6
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T08:34:27.265Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: FreeInit: Bridging Initialization Gap in Video Diffusion Models
paper_summary: Though diffusion-based video generation has witnessed rapid progress, the
  inference results of existing models still exhibit unsatisfactory temporal
  consistency and unnatural dynamics. In this paper, we delve deep into the noise
  initialization of video diffusion models, and discover an implicit
  training-inference gap that attributes to the unsatisfactory inference quality.
  Our key findings are: 1) the spatial-temporal frequency distribution of the
  initial latent at inference is intrinsically different from that for training,
  and 2) the denoising process is significantly influenced by the low-frequency
  components of the initial noise. Motivated by these observations, we propose a
  concise yet effective inference sampling strategy, FreeInit, which
  significantly improves temporal consistency of videos generated by diffusion
  models. Through iteratively refining the spatial-temporal low-frequency
  components of the initial latent during inference, FreeInit is able to
  compensate the initialization gap between training and inference, thus
  effectively improving the subject appearance and temporal consistency of
  generation results. Extensive experiments demonstrate that FreeInit
  consistently enhances the generation results of various text-to-video
  generation models without additional training.
paper_ai_summary: FreeInit addresses the temporal consistency and unnatural dynamics issues in diffusion-based video generation by refining spatial-temporal low-frequency components during inference, improving subject appearance and consistency.
paper_ai_keywords: diffusion-based video generation temporal consistency inference quality noise initialization spatial-temporal frequency distribution initial latent denoising process low-frequency components FreeInit