title: Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model
summary: This paper addresses an important problem of object addition for images with
  only text guidance. It is challenging because the new object must be integrated
  seamlessly into the image with consistent visual context, such as lighting,
  texture, and spatial location. While existing text-guided image inpainting
  methods can add objects, they either fail to preserve the background
  consistency or involve cumbersome human intervention in specifying bounding
  boxes or user-scribbled masks. To tackle this challenge, we introduce Diffree,
  a Text-to-Image (T2I) model that facilitates text-guided object addition with
  only text control. To this end, we curate OABench, an exquisite synthetic
  dataset by removing objects with advanced image inpainting techniques. OABench
  comprises 74K real-world tuples of an original image, an inpainted image with
  the object removed, an object mask, and object descriptions. Trained on OABench
  using the Stable Diffusion model with an additional mask prediction module,
  Diffree uniquely predicts the position of the new object and achieves object
  addition with guidance from only text. Extensive experiments demonstrate that
  Diffree excels in adding new objects with a high success rate while maintaining
  background consistency, spatial appropriateness, and object relevance and
  quality.
paper_authors._id: 66a1dad004ab5290ec1b405e
paper_authors.name: Lirui Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-25T07:38:19.597Z
paper_authors._id: 661639c4dd9284ce5d361f0f
paper_authors.avatarUrl: /avatars/9a0f8f725d7b90c30dfdb64313761155.svg
paper_authors.fullname: Lirui Zhao
paper_authors.type: user
paper_authors.user: LiruiZhao
paper_authors._id: 66a1dad004ab5290ec1b405f
paper_authors.name: Tianshuo Yang
paper_authors._id: 66a1dad004ab5290ec1b4060
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T07:58:24.207Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 66a1dad004ab5290ec1b4061
paper_authors.name: Yuxin Zhang
paper_authors._id: 66a1dad004ab5290ec1b4062
paper_authors.name: Yu Qiao
paper_authors._id: 66a1dad004ab5290ec1b4063
paper_authors.name: Ping Luo
paper_authors._id: 66a1dad004ab5290ec1b4064
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T08:06:11.679Z
paper_authors._id: 63527f4e7d071f23d085ad45
paper_authors.avatarUrl: /avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
paper_authors.fullname: KAIPENG ZHANG
paper_authors.type: user
paper_authors.user: kpzhang
paper_authors._id: 66a1dad004ab5290ec1b4065
paper_authors.name: Rongrong Ji
paper_title: Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model
paper_summary: This paper addresses an important problem of object addition for images with
  only text guidance. It is challenging because the new object must be integrated
  seamlessly into the image with consistent visual context, such as lighting,
  texture, and spatial location. While existing text-guided image inpainting
  methods can add objects, they either fail to preserve the background
  consistency or involve cumbersome human intervention in specifying bounding
  boxes or user-scribbled masks. To tackle this challenge, we introduce Diffree,
  a Text-to-Image (T2I) model that facilitates text-guided object addition with
  only text control. To this end, we curate OABench, an exquisite synthetic
  dataset by removing objects with advanced image inpainting techniques. OABench
  comprises 74K real-world tuples of an original image, an inpainted image with
  the object removed, an object mask, and object descriptions. Trained on OABench
  using the Stable Diffusion model with an additional mask prediction module,
  Diffree uniquely predicts the position of the new object and achieves object
  addition with guidance from only text. Extensive experiments demonstrate that
  Diffree excels in adding new objects with a high success rate while maintaining
  background consistency, spatial appropriateness, and object relevance and
  quality.
paper_ai_summary: Diffree, a Text-to-Image model using Stable Diffusion with a mask prediction module, achieves the addition of text-guided objects while maintaining background consistency and spatial appropriateness.
paper_ai_keywords: Diffree Text-to-Image (T2I) model Stable Diffusion model mask prediction module object addition background consistency spatial appropriateness object relevance quality OABench synthetic dataset image inpainting techniques object mask object descriptions