title: Audiobox: Unified Audio Generation with Natural Language Prompts
summary: Audio is an essential part of our life, but creating it often requires
  expertise and is time-consuming. Research communities have made great progress
  over the past year advancing the performance of large scale audio generative
  models for a single modality (speech, sound, or music) through adopting more
  powerful generative models and scaling data. However, these models lack
  controllability in several aspects: speech generation models cannot synthesize
  novel styles based on text description and are limited on domain coverage such
  as outdoor environments; sound generation models only provide coarse-grained
  control based on descriptions like "a person speaking" and would only generate
  mumbling human voices. This paper presents Audiobox, a unified model based on
  flow-matching that is capable of generating various audio modalities. We design
  description-based and example-based prompting to enhance controllability and
  unify speech and sound generation paradigms. We allow transcript, vocal, and
  other audio styles to be controlled independently when generating speech. To
  improve model generalization with limited labels, we adapt a self-supervised
  infilling objective to pre-train on large quantities of unlabeled audio.
  Audiobox sets new benchmarks on speech and sound generation (0.745 similarity
  on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and
  unlocks new methods for generating audio with novel vocal and acoustic styles.
  We further integrate Bespoke Solvers, which speeds up generation by over 25
  times compared to the default ODE solver for flow-matching, without loss of
  performance on several tasks. Our demo is available at
  https://audiobox.metademolab.com/
paper_authors._id: 658bb8df1783d1ebc7639690
paper_authors.name: Apoorv Vyas
paper_authors._id: 658bb8df1783d1ebc7639691
paper_authors.name: Bowen Shi
paper_authors._id: 658bb8df1783d1ebc7639692
paper_authors.name: Matthew Le
paper_authors._id: 658bb8df1783d1ebc7639693
paper_authors.name: Andros Tjandra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T20:36:52.153Z
paper_authors._id: 647779c6c21496284db45835
paper_authors.avatarUrl: /avatars/c1176e57f5d0aaa5f50bbde48a19af01.svg
paper_authors.fullname: Andros Tjandra
paper_authors.type: user
paper_authors.user: androstj
paper_authors._id: 658bb8df1783d1ebc7639694
paper_authors.name: Yi-Chiao Wu
paper_authors._id: 658bb8df1783d1ebc7639695
paper_authors.name: Baishan Guo
paper_authors._id: 658bb8df1783d1ebc7639696
paper_authors.name: Jiemin Zhang
paper_authors._id: 658bb8df1783d1ebc7639697
paper_authors.name: Xinyue Zhang
paper_authors._id: 658bb8df1783d1ebc7639698
paper_authors.name: Robert Adkins
paper_authors._id: 658bb8df1783d1ebc7639699
paper_authors.name: William Ngan
paper_authors._id: 658bb8df1783d1ebc763969a
paper_authors.name: Jeff Wang
paper_authors._id: 658bb8df1783d1ebc763969b
paper_authors.name: Ivan Cruz
paper_authors._id: 658bb8df1783d1ebc763969c
paper_authors.name: Bapi Akula
paper_authors._id: 658bb8df1783d1ebc763969d
paper_authors.name: Akinniyi Akinyemi
paper_authors._id: 658bb8df1783d1ebc763969e
paper_authors.name: Brian Ellis
paper_authors._id: 658bb8df1783d1ebc763969f
paper_authors.name: Rashel Moritz
paper_authors._id: 658bb8df1783d1ebc76396a0
paper_authors.name: Yael Yungster
paper_authors._id: 658bb8df1783d1ebc76396a1
paper_authors.name: Alice Rakotoarison
paper_authors._id: 658bb8df1783d1ebc76396a2
paper_authors.name: Liang Tan
paper_authors._id: 658bb8df1783d1ebc76396a3
paper_authors.name: Chris Summers
paper_authors._id: 658bb8df1783d1ebc76396a4
paper_authors.name: Carleigh Wood
paper_authors._id: 658bb8df1783d1ebc76396a5
paper_authors.name: Joshua Lane
paper_authors._id: 658bb8df1783d1ebc76396a6
paper_authors.name: Mary Williamson
paper_authors._id: 658bb8df1783d1ebc76396a7
paper_authors.name: Wei-Ning Hsu
paper_title: Audiobox: Unified Audio Generation with Natural Language Prompts
paper_summary: Audio is an essential part of our life, but creating it often requires
  expertise and is time-consuming. Research communities have made great progress
  over the past year advancing the performance of large scale audio generative
  models for a single modality (speech, sound, or music) through adopting more
  powerful generative models and scaling data. However, these models lack
  controllability in several aspects: speech generation models cannot synthesize
  novel styles based on text description and are limited on domain coverage such
  as outdoor environments; sound generation models only provide coarse-grained
  control based on descriptions like "a person speaking" and would only generate
  mumbling human voices. This paper presents Audiobox, a unified model based on
  flow-matching that is capable of generating various audio modalities. We design
  description-based and example-based prompting to enhance controllability and
  unify speech and sound generation paradigms. We allow transcript, vocal, and
  other audio styles to be controlled independently when generating speech. To
  improve model generalization with limited labels, we adapt a self-supervised
  infilling objective to pre-train on large quantities of unlabeled audio.
  Audiobox sets new benchmarks on speech and sound generation (0.745 similarity
  on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and
  unlocks new methods for generating audio with novel vocal and acoustic styles.
  We further integrate Bespoke Solvers, which speeds up generation by over 25
  times compared to the default ODE solver for flow-matching, without loss of
  performance on several tasks. Our demo is available at
  https://audiobox.metademolab.com/
paper_ai_summary: Audiobox, a flow-matching based unified model, enhances controllability and efficiency in generating various audio modalities including speech and sound using description-based and example-based prompting.
paper_ai_keywords: flow-matching description-based prompting example-based prompting speech generation sound generation self-supervised infilling zero-shot TTS text-to-sound Librispeech AudioCaps Bespoke Solvers ODE solver