title: Video Creation by Demonstration
summary: We explore a novel video creation experience, namely Video Creation by
  Demonstration. Given a demonstration video and a context image from a different
  scene, we generate a physically plausible video that continues naturally from
  the context image and carries out the action concepts from the demonstration.
  To enable this capability, we present delta-Diffusion, a self-supervised
  training approach that learns from unlabeled videos by conditional future frame
  prediction. Unlike most existing video generation controls that are based on
  explicit signals, we adopts the form of implicit latent control for maximal
  flexibility and expressiveness required by general videos. By leveraging a
  video foundation model with an appearance bottleneck design on top, we extract
  action latents from demonstration videos for conditioning the generation
  process with minimal appearance leakage. Empirically, delta-Diffusion
  outperforms related baselines in terms of both human preference and large-scale
  machine evaluations, and demonstrates potentials towards interactive world
  simulation. Sampled video generation results are available at
  https://delta-diffusion.github.io/.
paper_authors._id: 675b963adbb389af011b5a01
paper_authors.name: Yihong Sun
paper_authors._id: 675b963adbb389af011b5a02
paper_authors.name: Hao Zhou
paper_authors._id: 675b963adbb389af011b5a03
paper_authors.name: Liangzhe Yuan
paper_authors._id: 675b963adbb389af011b5a04
paper_authors.name: Jennifer J. Sun
paper_authors._id: 675b963adbb389af011b5a05
paper_authors.name: Yandong Li
paper_authors._id: 675b963adbb389af011b5a06
paper_authors.name: Xuhui Jia
paper_authors._id: 675b963adbb389af011b5a07
paper_authors.name: Hartwig Adam
paper_authors._id: 675b963adbb389af011b5a08
paper_authors.name: Bharath Hariharan
paper_authors._id: 675b963adbb389af011b5a09
paper_authors.name: Long Zhao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-13T02:13:07.970Z
paper_authors._id: 650c249887dcda6616baa040
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tIVfxoAHAJ0sWyDMkaarA.jpeg
paper_authors.fullname: Long Zhao
paper_authors.type: user
paper_authors.user: garyzhao9012
paper_authors._id: 675b963adbb389af011b5a0a
paper_authors.name: Ting Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:35.393Z
paper_authors._id: 64a78e9bacc69c3bbaf12b3d
paper_authors.avatarUrl: /avatars/09cd26234e0e0c82f5cc2c02e710163c.svg
paper_authors.fullname: Ting Liu
paper_authors.type: user
paper_authors.user: tingianliu
paper_title: Video Creation by Demonstration
paper_summary: We explore a novel video creation experience, namely Video Creation by
  Demonstration. Given a demonstration video and a context image from a different
  scene, we generate a physically plausible video that continues naturally from
  the context image and carries out the action concepts from the demonstration.
  To enable this capability, we present delta-Diffusion, a self-supervised
  training approach that learns from unlabeled videos by conditional future frame
  prediction. Unlike most existing video generation controls that are based on
  explicit signals, we adopts the form of implicit latent control for maximal
  flexibility and expressiveness required by general videos. By leveraging a
  video foundation model with an appearance bottleneck design on top, we extract
  action latents from demonstration videos for conditioning the generation
  process with minimal appearance leakage. Empirically, delta-Diffusion
  outperforms related baselines in terms of both human preference and large-scale
  machine evaluations, and demonstrates potentials towards interactive world
  simulation. Sampled video generation results are available at
  https://delta-diffusion.github.io/.
paper_ai_keywords: $\delta$-Diffusion self-supervised training conditional future frame prediction implicit latent control video foundation model appearance bottleneck design action latents human preference large-scale machine evaluations interactive world simulation