title: ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object
    Removal and Insertion
summary: Diffusion models have revolutionized image editing but often generate images
  that violate physical laws, particularly the effects of objects on the scene,
  e.g., occlusions, shadows, and reflections. By analyzing the limitations of
  self-supervised approaches, we propose a practical solution centered on a
  counterfactual dataset. Our method involves capturing a scene before and
  after removing a single object, while minimizing other changes. By fine-tuning
  a diffusion model on this dataset, we are able to not only remove objects but
  also their effects on the scene. However, we find that applying this approach
  for photorealistic object insertion requires an impractically large dataset. To
  tackle this challenge, we propose bootstrap supervision; leveraging our object
  removal model trained on a small counterfactual dataset, we synthetically
  expand this dataset considerably. Our approach significantly outperforms prior
  methods in photorealistic object removal and insertion, particularly at
  modeling the effects of objects on the scene.
paper_authors._id: 6604e3cf9aad5f4dcd998755
paper_authors.name: Daniel Winter
paper_authors._id: 6604e3cf9aad5f4dcd998756
paper_authors.name: Matan Cohen
paper_authors._id: 6604e3cf9aad5f4dcd998757
paper_authors.name: Shlomi Fruchter
paper_authors._id: 6604e3cf9aad5f4dcd998758
paper_authors.name: Yael Pritch
paper_authors._id: 6604e3cf9aad5f4dcd998759
paper_authors.name: Alex Rav-Acha
paper_authors._id: 6604e3cf9aad5f4dcd99875a
paper_authors.name: Yedid Hoshen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:43:55.377Z
paper_authors._id: 646cfc3b4220471ca0c56b20
paper_authors.avatarUrl: /avatars/19d6ab141ec2cd25c1c3b45fd8f69910.svg
paper_authors.fullname: Yedid Hoshen
paper_authors.type: user
paper_authors.user: yedid
paper_title: ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object
    Removal and Insertion
paper_summary: Diffusion models have revolutionized image editing but often generate images
  that violate physical laws, particularly the effects of objects on the scene,
  e.g., occlusions, shadows, and reflections. By analyzing the limitations of
  self-supervised approaches, we propose a practical solution centered on a
  counterfactual dataset. Our method involves capturing a scene before and
  after removing a single object, while minimizing other changes. By fine-tuning
  a diffusion model on this dataset, we are able to not only remove objects but
  also their effects on the scene. However, we find that applying this approach
  for photorealistic object insertion requires an impractically large dataset. To
  tackle this challenge, we propose bootstrap supervision; leveraging our object
  removal model trained on a small counterfactual dataset, we synthetically
  expand this dataset considerably. Our approach significantly outperforms prior
  methods in photorealistic object removal and insertion, particularly at
  modeling the effects of objects on the scene.
paper_ai_keywords: diffusion models counterfactual dataset fine-tuning photorealistic object removal object insertion bootstrap supervision synthetic expansion