title: ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation
summary: Image-to-video (I2V) generation aims to use the initial frame (alongside a
  text prompt) to create a video sequence. A grand challenge in I2V generation is
  to maintain visual consistency throughout the video: existing methods often
  struggle to preserve the integrity of the subject, background, and style from
  the first frame, as well as ensure a fluid and logical progression within the
  video narrative. To mitigate these issues, we propose ConsistI2V, a
  diffusion-based method to enhance visual consistency for I2V generation.
  Specifically, we introduce (1) spatiotemporal attention over the first frame to
  maintain spatial and motion consistency, (2) noise initialization from the
  low-frequency band of the first frame to enhance layout consistency. These two
  approaches enable ConsistI2V to generate highly consistent videos. We also
  extend the proposed approaches to show their potential to improve consistency
  in auto-regressive long video generation and camera motion control. To verify
  the effectiveness of our method, we propose I2V-Bench, a comprehensive
  evaluation benchmark for I2V generation. Our automatic and human evaluation
  results demonstrate the superiority of ConsistI2V over existing methods.
paper_authors._id: 65c439807fe765199aca09d8
paper_authors.name: Weiming Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-08T09:50:08.838Z
paper_authors._id: 64405a9d518271b0d1beea38
paper_authors.avatarUrl: /avatars/b702474588fd7090773320422417a582.svg
paper_authors.fullname: Weiming Ren
paper_authors.type: user
paper_authors.user: wren93
paper_authors._id: 65c439807fe765199aca09d9
paper_authors.name: Harry Yang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-08T11:30:24.015Z
paper_authors._id: 639005861c5a6237272557d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670382915881-noauth.jpeg
paper_authors.fullname: Huan Yang
paper_authors.type: user
paper_authors.user: hyang0511
paper_authors._id: 65c439807fe765199aca09da
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-08T09:50:10.959Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 65c439807fe765199aca09db
paper_authors.name: Cong Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-08T09:50:12.718Z
paper_authors._id: 64f8e358766ff9f3d2b0de84
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f8e358766ff9f3d2b0de84/R2P1YG-mRBh7TU9wkjGGk.jpeg
paper_authors.fullname: Cong Wei
paper_authors.type: user
paper_authors.user: lim142857
paper_authors._id: 65c439807fe765199aca09dc
paper_authors.name: Xinrun Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:54:15.529Z
paper_authors._id: 654907a4a1faff97850c4eff
paper_authors.avatarUrl: /avatars/458c90151614bc7f116943b6e67d6b8a.svg
paper_authors.fullname: du
paper_authors.type: user
paper_authors.user: dododododo
paper_authors._id: 65c439807fe765199aca09dd
paper_authors.name: Stephen Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:25:39.334Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 65c439807fe765199aca09de
paper_authors.name: Wenhu Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-08T02:16:36.951Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation
paper_summary: Image-to-video (I2V) generation aims to use the initial frame (alongside a
  text prompt) to create a video sequence. A grand challenge in I2V generation is
  to maintain visual consistency throughout the video: existing methods often
  struggle to preserve the integrity of the subject, background, and style from
  the first frame, as well as ensure a fluid and logical progression within the
  video narrative. To mitigate these issues, we propose ConsistI2V, a
  diffusion-based method to enhance visual consistency for I2V generation.
  Specifically, we introduce (1) spatiotemporal attention over the first frame to
  maintain spatial and motion consistency, (2) noise initialization from the
  low-frequency band of the first frame to enhance layout consistency. These two
  approaches enable ConsistI2V to generate highly consistent videos. We also
  extend the proposed approaches to show their potential to improve consistency
  in auto-regressive long video generation and camera motion control. To verify
  the effectiveness of our method, we propose I2V-Bench, a comprehensive
  evaluation benchmark for I2V generation. Our automatic and human evaluation
  results demonstrate the superiority of ConsistI2V over existing methods.
paper_ai_keywords: diffusion-based method spatiotemporal attention noise initialization low-frequency band layout consistency I2V-Bench I2V generation video narrative visual consistency spatial consistency motion consistency auto-regressive long video generation camera motion control