title: Paint by Inpaint: Learning to Add Image Objects by Removing Them First
summary: Image editing has advanced significantly with the introduction of
  text-conditioned diffusion models. Despite this progress, seamlessly adding
  objects to images based on textual instructions without requiring user-provided
  input masks remains a challenge. We address this by leveraging the insight that
  removing objects (Inpaint) is significantly simpler than its inverse process of
  adding them (Paint), attributed to the utilization of segmentation mask
  datasets alongside inpainting models that inpaint within these masks.
  Capitalizing on this realization, by implementing an automated and extensive
  pipeline, we curate a filtered large-scale image dataset containing pairs of
  images and their corresponding object-removed versions. Using these pairs, we
  train a diffusion model to inverse the inpainting process, effectively adding
  objects into images. Unlike other editing datasets, ours features natural
  target images instead of synthetic ones; moreover, it maintains consistency
  between source and target by construction. Additionally, we utilize a large
  Vision-Language Model to provide detailed descriptions of the removed objects
  and a Large Language Model to convert these descriptions into diverse,
  natural-language instructions. We show that the trained model surpasses
  existing ones both qualitatively and quantitatively, and release the
  large-scale dataset alongside the trained models for the community.
paper_authors._id: 66308f08f0879eea6d52eb6f
paper_authors.name: Navve Wasserman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:32:07.764Z
paper_authors._id: 6492c419702103104f9450c4
paper_authors.avatarUrl: /avatars/d6700e4bb1b9f172096ea31ba15a83b2.svg
paper_authors.fullname: navve wasserman
paper_authors.type: user
paper_authors.user: navvew
paper_authors._id: 66308f08f0879eea6d52eb70
paper_authors.name: Noam Rotstein
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-30T07:27:18.700Z
paper_authors._id: 62b3e85bcbd2a402fc7804b1
paper_authors.avatarUrl: /avatars/63125ce8a1e20b8c6e836f223d24284f.svg
paper_authors.fullname: noam rotstein
paper_authors.type: user
paper_authors.user: noamrot
paper_authors._id: 66308f08f0879eea6d52eb71
paper_authors.name: Roy Ganz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:32:14.715Z
paper_authors._id: 62f0cb39671fc964b5063aba
paper_authors.avatarUrl: /avatars/02404cd78c395331b42500dd6ced35eb.svg
paper_authors.fullname: Roy Ganz
paper_authors.type: user
paper_authors.user: proy
paper_authors._id: 66308f08f0879eea6d52eb72
paper_authors.name: Ron Kimmel
paper_title: Paint by Inpaint: Learning to Add Image Objects by Removing Them First
paper_summary: Image editing has advanced significantly with the introduction of
  text-conditioned diffusion models. Despite this progress, seamlessly adding
  objects to images based on textual instructions without requiring user-provided
  input masks remains a challenge. We address this by leveraging the insight that
  removing objects (Inpaint) is significantly simpler than its inverse process of
  adding them (Paint), attributed to the utilization of segmentation mask
  datasets alongside inpainting models that inpaint within these masks.
  Capitalizing on this realization, by implementing an automated and extensive
  pipeline, we curate a filtered large-scale image dataset containing pairs of
  images and their corresponding object-removed versions. Using these pairs, we
  train a diffusion model to inverse the inpainting process, effectively adding
  objects into images. Unlike other editing datasets, ours features natural
  target images instead of synthetic ones; moreover, it maintains consistency
  between source and target by construction. Additionally, we utilize a large
  Vision-Language Model to provide detailed descriptions of the removed objects
  and a Large Language Model to convert these descriptions into diverse,
  natural-language instructions. We show that the trained model surpasses
  existing ones both qualitatively and quantitatively, and release the
  large-scale dataset alongside the trained models for the community.
paper_ai_summary: A diffusion model is trained to add objects to images based on text descriptions using a large-scale dataset of images with removed objects, outperforming existing methods.
paper_ai_keywords: text-conditioned diffusion models inpainting object removal segmentation mask datasets Vision-Language Model Large Language Model dataset curation inverse inpainting natural target images source-target consistency qualitative and quantitative evaluation