Paint by Inpaint: Learning to Add Image Objects by Removing Them First
Image editing has advanced significantly with the introduction of
text-conditioned diffusion models. Despite this progress, seamlessly adding
objects to images based on textual instructions without requiring user-provided
input masks remains a challenge. We address this by leveraging the insight that
removing objects (Inpaint) is significantly simpler than its inverse process of
adding them (Paint), attributed to the utilization of segmentation mask
datasets alongside inpainting models that inpaint within these masks.
Capitalizing on this realization, by implementing an automated and extensive
pipeline, we curate a filtered large-scale image dataset containing pairs of
images and their corresponding object-removed versions. Using these pairs, we
train a diffusion model to inverse the inpainting process, effectively adding
objects into images. Unlike other editing datasets, ours features natural
target images instead of synthetic ones; moreover, it maintains consistency
between source and target by construction. Additionally, we utilize a large
Vision-Language Model to provide detailed descriptions of the removed objects
and a Large Language Model to convert these descriptions into diverse,
natural-language instructions. We show that the trained model surpasses
existing ones both qualitatively and quantitatively, and release the
large-scale dataset alongside the trained models for the community.
66308f08f0879eea6d52eb6f
Navve Wasserman
admin_assigned
2024-05-02T07:32:07.764Z
6492c419702103104f9450c4
/avatars/d6700e4bb1b9f172096ea31ba15a83b2.svg
navve wasserman
user
navvew
66308f08f0879eea6d52eb70
Noam Rotstein
claimed_verified
2024-04-30T07:27:18.700Z
62b3e85bcbd2a402fc7804b1
/avatars/63125ce8a1e20b8c6e836f223d24284f.svg
noam rotstein
user
noamrot
66308f08f0879eea6d52eb71
Roy Ganz
admin_assigned
2024-05-02T07:32:14.715Z
62f0cb39671fc964b5063aba
/avatars/02404cd78c395331b42500dd6ced35eb.svg
Roy Ganz
user
proy
66308f08f0879eea6d52eb72
Ron Kimmel
Paint by Inpaint: Learning to Add Image Objects by Removing Them First
Image editing has advanced significantly with the introduction of
text-conditioned diffusion models. Despite this progress, seamlessly adding
objects to images based on textual instructions without requiring user-provided
input masks remains a challenge. We address this by leveraging the insight that
removing objects (Inpaint) is significantly simpler than its inverse process of
adding them (Paint), attributed to the utilization of segmentation mask
datasets alongside inpainting models that inpaint within these masks.
Capitalizing on this realization, by implementing an automated and extensive
pipeline, we curate a filtered large-scale image dataset containing pairs of
images and their corresponding object-removed versions. Using these pairs, we
train a diffusion model to inverse the inpainting process, effectively adding
objects into images. Unlike other editing datasets, ours features natural
target images instead of synthetic ones; moreover, it maintains consistency
between source and target by construction. Additionally, we utilize a large
Vision-Language Model to provide detailed descriptions of the removed objects
and a Large Language Model to convert these descriptions into diverse,
natural-language instructions. We show that the trained model surpasses
existing ones both qualitatively and quantitatively, and release the
large-scale dataset alongside the trained models for the community.
A diffusion model is trained to add objects to images based on text descriptions using a large-scale dataset of images with removed objects, outperforming existing methods.
text-conditioned diffusion models inpainting object removal segmentation mask datasets Vision-Language Model Large Language Model dataset curation inverse inpainting natural target images source-target consistency qualitative and quantitative evaluation