title: VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing
summary: Recently, diffusion-based generative models have achieved remarkable success
  for image generation and edition. However, their use for video editing still
  faces important limitations. This paper introduces VidEdit, a novel method for
  zero-shot text-based video editing ensuring strong temporal and spatial
  consistency. Firstly, we propose to combine atlas-based and pre-trained
  text-to-image diffusion models to provide a training-free and efficient editing
  method, which by design fulfills temporal smoothness. Secondly, we leverage
  off-the-shelf panoptic segmenters along with edge detectors and adapt their use
  for conditioned diffusion-based atlas editing. This ensures a fine spatial
  control on targeted regions while strictly preserving the structure of the
  original video. Quantitative and qualitative experiments show that VidEdit
  outperforms state-of-the-art methods on DAVIS dataset, regarding semantic
  faithfulness, image preservation, and temporal consistency metrics. With this
  framework, processing a single video only takes approximately one minute, and
  it can generate multiple compatible edits based on a unique text prompt.
  Project web-page at https://videdit.github.io
paper_authors._id: 648beeec10bced401c50e780
paper_authors.name: Paul Couairon
paper_authors._id: 648beeec10bced401c50e781
paper_authors.name: Clément Rambour
paper_authors._id: 648beeec10bced401c50e782
paper_authors.name: Jean-Emmanuel Haugeard
paper_authors._id: 648beeec10bced401c50e783
paper_authors.name: Nicolas Thome
paper_title: VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing
paper_summary: Recently, diffusion-based generative models have achieved remarkable success
  for image generation and edition. However, their use for video editing still
  faces important limitations. This paper introduces VidEdit, a novel method for
  zero-shot text-based video editing ensuring strong temporal and spatial
  consistency. Firstly, we propose to combine atlas-based and pre-trained
  text-to-image diffusion models to provide a training-free and efficient editing
  method, which by design fulfills temporal smoothness. Secondly, we leverage
  off-the-shelf panoptic segmenters along with edge detectors and adapt their use
  for conditioned diffusion-based atlas editing. This ensures a fine spatial
  control on targeted regions while strictly preserving the structure of the
  original video. Quantitative and qualitative experiments show that VidEdit
  outperforms state-of-the-art methods on DAVIS dataset, regarding semantic
  faithfulness, image preservation, and temporal consistency metrics. With this
  framework, processing a single video only takes approximately one minute, and
  it can generate multiple compatible edits based on a unique text prompt.
  Project web-page at https://videdit.github.io
paper_ai_summary: VidEdit is a zero-shot text-based video editing method combining atlas-based and pre-trained diffusion models with panoptic segmenters and edge detectors to achieve strong temporal and spatial consistency.
paper_ai_keywords: diffusion-based generative models text-to-image diffusion models atlas-based editing panoptic segmenters edge detectors DAVIS dataset semantic faithfulness image preservation temporal consistency