title: ZeroSep: Separate Anything in Audio with Zero Training
summary: Audio source separation is fundamental for machines to understand complex
  acoustic environments and underpins numerous audio applications. Current
  supervised deep learning approaches, while powerful, are limited by the need
  for extensive, task-specific labeled data and struggle to generalize to the
  immense variability and open-set nature of real-world acoustic scenes. Inspired
  by the success of generative foundation models, we investigate whether
  pre-trained text-guided audio diffusion models can overcome these limitations.
  We make a surprising discovery: zero-shot source separation can be achieved
  purely through a pre-trained text-guided audio diffusion model under the right
  configuration. Our method, named ZeroSep, works by inverting the mixed audio
  into the diffusion model's latent space and then using text conditioning to
  guide the denoising process to recover individual sources. Without any
  task-specific training or fine-tuning, ZeroSep repurposes the generative
  diffusion model for a discriminative separation task and inherently supports
  open-set scenarios through its rich textual priors. ZeroSep is compatible with
  a variety of pre-trained text-guided audio diffusion backbones and delivers
  strong separation performance on multiple separation benchmarks, surpassing
  even supervised methods.
paper_authors._id: 683924d1ac00da416df936dd
paper_authors.name: Chao Huang
paper_authors._id: 683924d1ac00da416df936de
paper_authors.name: Yuesheng Ma
paper_authors._id: 683924d1ac00da416df936df
paper_authors.name: Junxuan Huang
paper_authors._id: 683924d1ac00da416df936e0
paper_authors.name: Susan Liang
paper_authors._id: 683924d1ac00da416df936e1
paper_authors.name: Yunlong Tang
paper_authors._id: 683924d1ac00da416df936e2
paper_authors.name: Jing Bi
paper_authors._id: 683924d1ac00da416df936e3
paper_authors.name: Wenqiang Liu
paper_authors._id: 683924d1ac00da416df936e4
paper_authors.name: Nima Mesgarani
paper_authors._id: 683924d1ac00da416df936e5
paper_authors.name: Chenliang Xu
paper_title: ZeroSep: Separate Anything in Audio with Zero Training
paper_summary: Audio source separation is fundamental for machines to understand complex
  acoustic environments and underpins numerous audio applications. Current
  supervised deep learning approaches, while powerful, are limited by the need
  for extensive, task-specific labeled data and struggle to generalize to the
  immense variability and open-set nature of real-world acoustic scenes. Inspired
  by the success of generative foundation models, we investigate whether
  pre-trained text-guided audio diffusion models can overcome these limitations.
  We make a surprising discovery: zero-shot source separation can be achieved
  purely through a pre-trained text-guided audio diffusion model under the right
  configuration. Our method, named ZeroSep, works by inverting the mixed audio
  into the diffusion model's latent space and then using text conditioning to
  guide the denoising process to recover individual sources. Without any
  task-specific training or fine-tuning, ZeroSep repurposes the generative
  diffusion model for a discriminative separation task and inherently supports
  open-set scenarios through its rich textual priors. ZeroSep is compatible with
  a variety of pre-trained text-guided audio diffusion backbones and delivers
  strong separation performance on multiple separation benchmarks, surpassing
  even supervised methods.
paper_ai_summary: ZeroSep, a text-guided audio diffusion model, achieves zero-shot source separation through pre-trained models and text conditioning, outperforming supervised methods on various benchmarks.
paper_ai_keywords: audio source separation supervised deep learning generative foundation models text-guided audio diffusion models zero-shot source separation latent space denoising process discriminative separation task textual priors open-set scenarios