Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model
This paper addresses an important problem of object addition for images with
only text guidance. It is challenging because the new object must be integrated
seamlessly into the image with consistent visual context, such as lighting,
texture, and spatial location. While existing text-guided image inpainting
methods can add objects, they either fail to preserve the background
consistency or involve cumbersome human intervention in specifying bounding
boxes or user-scribbled masks. To tackle this challenge, we introduce Diffree,
a Text-to-Image (T2I) model that facilitates text-guided object addition with
only text control. To this end, we curate OABench, an exquisite synthetic
dataset by removing objects with advanced image inpainting techniques. OABench
comprises 74K real-world tuples of an original image, an inpainted image with
the object removed, an object mask, and object descriptions. Trained on OABench
using the Stable Diffusion model with an additional mask prediction module,
Diffree uniquely predicts the position of the new object and achieves object
addition with guidance from only text. Extensive experiments demonstrate that
Diffree excels in adding new objects with a high success rate while maintaining
background consistency, spatial appropriateness, and object relevance and
quality.
66a1dad004ab5290ec1b405e
Lirui Zhao
claimed_verified
2024-07-25T07:38:19.597Z
661639c4dd9284ce5d361f0f
/avatars/9a0f8f725d7b90c30dfdb64313761155.svg
Lirui Zhao
user
LiruiZhao
66a1dad004ab5290ec1b405f
Tianshuo Yang
claimed_verified
2025-07-13T07:30:43.035Z
657010abf2c1c863093941cb
/avatars/25b8a1ecdea3b5ab687dc8331b5f8615.svg
Tianshuo
user
Violin-Y
66a1dad004ab5290ec1b4060
Wenqi Shao
admin_assigned
2024-07-26T07:58:24.207Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
66a1dad004ab5290ec1b4061
Yuxin Zhang
66a1dad004ab5290ec1b4062
Yu Qiao
66a1dad004ab5290ec1b4063
Ping Luo
66a1dad004ab5290ec1b4064
Kaipeng Zhang
admin_assigned
2024-07-26T08:06:11.679Z
63527f4e7d071f23d085ad45
/avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
KAIPENG ZHANG
user
kpzhang
66a1dad004ab5290ec1b4065
Rongrong Ji
Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model
This paper addresses an important problem of object addition for images with
only text guidance. It is challenging because the new object must be integrated
seamlessly into the image with consistent visual context, such as lighting,
texture, and spatial location. While existing text-guided image inpainting
methods can add objects, they either fail to preserve the background
consistency or involve cumbersome human intervention in specifying bounding
boxes or user-scribbled masks. To tackle this challenge, we introduce Diffree,
a Text-to-Image (T2I) model that facilitates text-guided object addition with
only text control. To this end, we curate OABench, an exquisite synthetic
dataset by removing objects with advanced image inpainting techniques. OABench
comprises 74K real-world tuples of an original image, an inpainted image with
the object removed, an object mask, and object descriptions. Trained on OABench
using the Stable Diffusion model with an additional mask prediction module,
Diffree uniquely predicts the position of the new object and achieves object
addition with guidance from only text. Extensive experiments demonstrate that
Diffree excels in adding new objects with a high success rate while maintaining
background consistency, spatial appropriateness, and object relevance and
quality.
Diffree, a Text-to-Image model using Stable Diffusion with a mask prediction module, achieves the addition of text-guided objects while maintaining background consistency and spatial appropriateness.
Diffree Text-to-Image (T2I) model Stable Diffusion model mask prediction module object addition background consistency spatial appropriateness object relevance quality OABench synthetic dataset image inpainting techniques object mask object descriptions