title: Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion
summary: As a common image editing operation, image composition involves integrating
  foreground objects into background scenes. In this paper, we expand the
  application of the concept of Affordance from human-centered image composition
  tasks to a more general object-scene composition framework, addressing the
  complex interplay between foreground objects and background scenes. Following
  the principle of Affordance, we define the affordance-aware object insertion
  task, which aims to seamlessly insert any object into any scene with various
  position prompts. To address the limited data issue and incorporate this task,
  we constructed the SAM-FB dataset, which contains over 3 million examples
  across more than 3,000 object categories. Furthermore, we propose the
  Mask-Aware Dual Diffusion (MADD) model, which utilizes a dual-stream
  architecture to simultaneously denoise the RGB image and the insertion mask. By
  explicitly modeling the insertion mask in the diffusion process, MADD
  effectively facilitates the notion of affordance. Extensive experimental
  results show that our method outperforms the state-of-the-art methods and
  exhibits strong generalization performance on in-the-wild images. Please refer
  to our code on https://github.com/KaKituken/affordance-aware-any.
paper_authors._id: 6764e0339aeafaa0d8405e74
paper_authors.name: Jixuan He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:24:47.947Z
paper_authors._id: 64a5bda1733731f38378fdd3
paper_authors.avatarUrl: /avatars/3ec7c553f92ced4ae0420ce16f7e71b9.svg
paper_authors.fullname: Jixuan HE
paper_authors.type: user
paper_authors.user: Kakituken
paper_authors._id: 6764e0339aeafaa0d8405e75
paper_authors.name: Wanhua Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:32:44.737Z
paper_authors._id: 658bb7e47459b6e471b9d2e6
paper_authors.avatarUrl: /avatars/efd8051b468b4dbcb5d149479de67c58.svg
paper_authors.fullname: Wanhua Li
paper_authors.type: user
paper_authors.user: EthanTaylor
paper_authors._id: 6764e0339aeafaa0d8405e76
paper_authors.name: Ye Liu
paper_authors._id: 6764e0339aeafaa0d8405e77
paper_authors.name: Junsik Kim
paper_authors._id: 6764e0339aeafaa0d8405e78
paper_authors.name: Donglai Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:25:00.011Z
paper_authors._id: 628c2c8ab80bb09700d6cb1d
paper_authors.avatarUrl: /avatars/384f75778fd5f07f249f2815a3039dca.svg
paper_authors.fullname: donglai wei
paper_authors.type: user
paper_authors.user: dwei
paper_authors._id: 6764e0339aeafaa0d8405e79
paper_authors.name: Hanspeter Pfister
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:24:54.200Z
paper_authors._id: 62acc69e36f7c7b7f65fccca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62acc69e36f7c7b7f65fccca/S8o0XE6TaQwLU8q3QPkct.png
paper_authors.fullname: Hanspeter Pfister
paper_authors.type: user
paper_authors.user: hpfister
paper_title: Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion
paper_summary: As a common image editing operation, image composition involves integrating
  foreground objects into background scenes. In this paper, we expand the
  application of the concept of Affordance from human-centered image composition
  tasks to a more general object-scene composition framework, addressing the
  complex interplay between foreground objects and background scenes. Following
  the principle of Affordance, we define the affordance-aware object insertion
  task, which aims to seamlessly insert any object into any scene with various
  position prompts. To address the limited data issue and incorporate this task,
  we constructed the SAM-FB dataset, which contains over 3 million examples
  across more than 3,000 object categories. Furthermore, we propose the
  Mask-Aware Dual Diffusion (MADD) model, which utilizes a dual-stream
  architecture to simultaneously denoise the RGB image and the insertion mask. By
  explicitly modeling the insertion mask in the diffusion process, MADD
  effectively facilitates the notion of affordance. Extensive experimental
  results show that our method outperforms the state-of-the-art methods and
  exhibits strong generalization performance on in-the-wild images. Please refer
  to our code on https://github.com/KaKituken/affordance-aware-any.
paper_ai_keywords: Affordance affordance-aware object insertion SAM-FB dataset Mask-Aware Dual Diffusion (MADD) model dual-stream architecture denoising diffusion process