title: OmnimatteZero: Training-free Real-time Omnimatte with Pre-trained Video
    Diffusion Models
summary: Omnimatte aims to decompose a given video into semantically meaningful
  layers, including the background and individual objects along with their
  associated effects, such as shadows and reflections. Existing methods often
  require extensive training or costly self-supervised optimization. In this
  paper, we present OmnimatteZero, a training-free approach that leverages
  off-the-shelf pre-trained video diffusion models for omnimatte. It can remove
  objects from videos, extract individual object layers along with their effects,
  and composite those objects onto new videos. We accomplish this by adapting
  zero-shot image inpainting techniques for video object removal, a task they
  fail to handle effectively out-of-the-box. We then show that self-attention
  maps capture information about the object and its footprints and use them to
  inpaint the object's effects, leaving a clean background. Additionally, through
  simple latent arithmetic, object layers can be isolated and recombined
  seamlessly with new video layers to produce new videos. Evaluations show that
  OmnimatteZero not only achieves superior performance in terms of background
  reconstruction but also sets a new record for the fastest Omnimatte approach,
  achieving real-time performance with minimal frame runtime.
paper_authors._id: 67e2706af6cf2764a534d4a5
paper_authors.name: Dvir Samuel
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-25T09:00:01.542Z
paper_authors._id: 630f0d48982455e61cc4cc08
paper_authors.avatarUrl: /avatars/eea6ed2e112e830effa98a4661c5474f.svg
paper_authors.fullname: Samuel
paper_authors.type: user
paper_authors.user: Dvir
paper_authors._id: 67e2706af6cf2764a534d4a6
paper_authors.name: Matan Levy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:25:56.024Z
paper_authors._id: 66633be10875aaaa9153c963
paper_authors.avatarUrl: /avatars/f47aaaf7b029ad3e99f49676a8f9a479.svg
paper_authors.fullname: Matan Levy
paper_authors.type: user
paper_authors.user: m98levy
paper_authors._id: 67e2706af6cf2764a534d4a7
paper_authors.name: Nir Darshan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:49:13.626Z
paper_authors._id: 67e420bb2c327e08f6b07657
paper_authors.avatarUrl: /avatars/d82b4dd21141b9eb92b2c3cff74f342d.svg
paper_authors.fullname: Nir Darshan
paper_authors.type: user
paper_authors.user: nirdrs
paper_authors._id: 67e2706af6cf2764a534d4a8
paper_authors.name: Gal Chechik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:26:05.156Z
paper_authors._id: 6493393f357b252af72196c5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6493393f357b252af72196c5/EWSy18XRcMRa_4XMM3Fu-.jpeg
paper_authors.fullname: Gal Chechik
paper_authors.type: user
paper_authors.user: galchechik
paper_authors._id: 67e2706af6cf2764a534d4a9
paper_authors.name: Rami Ben-Ari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:26:11.853Z
paper_authors._id: 64c5f22c2581696666ebed88
paper_authors.avatarUrl: /avatars/e85cd2d82f16ec10cad2b63929b2f05a.svg
paper_authors.fullname: Rami Ben-Ari
paper_authors.type: user
paper_authors.user: ramiben
paper_title: OmnimatteZero: Training-free Real-time Omnimatte with Pre-trained Video
    Diffusion Models
paper_summary: Omnimatte aims to decompose a given video into semantically meaningful
  layers, including the background and individual objects along with their
  associated effects, such as shadows and reflections. Existing methods often
  require extensive training or costly self-supervised optimization. In this
  paper, we present OmnimatteZero, a training-free approach that leverages
  off-the-shelf pre-trained video diffusion models for omnimatte. It can remove
  objects from videos, extract individual object layers along with their effects,
  and composite those objects onto new videos. We accomplish this by adapting
  zero-shot image inpainting techniques for video object removal, a task they
  fail to handle effectively out-of-the-box. We then show that self-attention
  maps capture information about the object and its footprints and use them to
  inpaint the object's effects, leaving a clean background. Additionally, through
  simple latent arithmetic, object layers can be isolated and recombined
  seamlessly with new video layers to produce new videos. Evaluations show that
  OmnimatteZero not only achieves superior performance in terms of background
  reconstruction but also sets a new record for the fastest Omnimatte approach,
  achieving real-time performance with minimal frame runtime.
paper_ai_summary: OmnimatteZero leverages pre-trained video diffusion models and zero-shot image inpainting to efficiently decompose videos into meaningful layers and perform object removal and recombination with real-time performance.
paper_ai_keywords: video diffusion models zero-shot image inpainting self-attention maps latent arithmetic object removal video compositing