title: Sounding that Object: Interactive Object-Aware Image to Audio Generation
summary: Generating accurate sounds for complex audio-visual scenes is challenging,
  especially in the presence of multiple objects and sound sources. In this
  paper, we propose an {\em interactive object-aware audio generation} model that
  grounds sound generation in user-selected visual objects within images. Our
  method integrates object-centric learning into a conditional latent diffusion
  model, which learns to associate image regions with their corresponding sounds
  through multi-modal attention. At test time, our model employs image
  segmentation to allow users to interactively generate sounds at the {\em
  object} level. We theoretically validate that our attention mechanism
  functionally approximates test-time segmentation masks, ensuring the generated
  audio aligns with selected objects. Quantitative and qualitative evaluations
  show that our model outperforms baselines, achieving better alignment between
  objects and their associated sounds. Project page:
  https://tinglok.netlify.app/files/avobject/
paper_authors._id: 6841b0a2d198c64464545ae9
paper_authors.name: Tingle Li
paper_authors._id: 6841b0a2d198c64464545aea
paper_authors.name: Baihe Huang
paper_authors._id: 6841b0a2d198c64464545aeb
paper_authors.name: Xiaobin Zhuang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T15:07:43.456Z
paper_authors._id: 63774ca43a63a2983ffc12f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63774ca43a63a2983ffc12f9/jUJasGf5_2rwiEun_eYl7.png
paper_authors.fullname: xiaobin zhuang
paper_authors.type: user
paper_authors.user: xiaobinzhuang
paper_authors._id: 6841b0a2d198c64464545aec
paper_authors.name: Dongya Jia
paper_authors._id: 6841b0a2d198c64464545aed
paper_authors.name: Jiawei Chen
paper_authors._id: 6841b0a2d198c64464545aee
paper_authors.name: Yuping Wang
paper_authors._id: 6841b0a2d198c64464545aef
paper_authors.name: Zhuo Chen
paper_authors._id: 6841b0a2d198c64464545af0
paper_authors.name: Gopala Anumanchipalli
paper_authors._id: 6841b0a2d198c64464545af1
paper_authors.name: Yuxuan Wang
paper_title: Sounding that Object: Interactive Object-Aware Image to Audio Generation
paper_summary: Generating accurate sounds for complex audio-visual scenes is challenging,
  especially in the presence of multiple objects and sound sources. In this
  paper, we propose an {\em interactive object-aware audio generation} model that
  grounds sound generation in user-selected visual objects within images. Our
  method integrates object-centric learning into a conditional latent diffusion
  model, which learns to associate image regions with their corresponding sounds
  through multi-modal attention. At test time, our model employs image
  segmentation to allow users to interactively generate sounds at the {\em
  object} level. We theoretically validate that our attention mechanism
  functionally approximates test-time segmentation masks, ensuring the generated
  audio aligns with selected objects. Quantitative and qualitative evaluations
  show that our model outperforms baselines, achieving better alignment between
  objects and their associated sounds. Project page:
  https://tinglok.netlify.app/files/avobject/
paper_ai_summary: An interactive object-aware audio generation model aligns sounds with user-selected visual objects in images using a conditional latent diffusion model with multi-modal attention.
paper_ai_keywords: interactive object-aware audio generation object-centric learning conditional latent diffusion model multi-modal attention image segmentation quantitative evaluation qualitative evaluation