title: SoundCTM: Uniting Score-based and Consistency Models for Text-to-Sound
    Generation
summary: Sound content is an indispensable element for multimedia works such as video
  games, music, and films. Recent high-quality diffusion-based sound generation
  models can serve as valuable tools for the creators. However, despite producing
  high-quality sounds, these models often suffer from slow inference speeds. This
  drawback burdens creators, who typically refine their sounds through trial and
  error to align them with their artistic intentions. To address this issue, we
  introduce Sound Consistency Trajectory Models (SoundCTM). Our model enables
  flexible transitioning between high-quality 1-step sound generation and
  superior sound quality through multi-step generation. This allows creators to
  initially control sounds with 1-step samples before refining them through
  multi-step generation. While CTM fundamentally achieves flexible 1-step and
  multi-step generation, its impressive performance heavily depends on an
  additional pretrained feature extractor and an adversarial loss, which are
  expensive to train and not always available in other domains. Thus, we reframe
  CTM's training framework and introduce a novel feature distance by utilizing
  the teacher's network for a distillation loss. Additionally, while distilling
  classifier-free guided trajectories, we train conditional and unconditional
  student models simultaneously and interpolate between these models during
  inference. We also propose training-free controllable frameworks for SoundCTM,
  leveraging its flexible sampling capability. SoundCTM achieves both promising
  1-step and multi-step real-time sound generation without using any extra
  off-the-shelf networks. Furthermore, we demonstrate SoundCTM's capability of
  controllable sound generation in a training-free manner.
paper_authors._id: 66588e3daad054f64123df03
paper_authors.name: Koichi Saito
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-04T19:56:16.447Z
paper_authors._id: 665f7129b75fa1b1a705b7ce
paper_authors.avatarUrl: /avatars/f9c8245c0441dc428edcefce8989444e.svg
paper_authors.fullname: Koichi Saito
paper_authors.type: user
paper_authors.user: koichisaito
paper_authors._id: 66588e3daad054f64123df04
paper_authors.name: Dongjun Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:27:31.977Z
paper_authors._id: 65dc46602bc96e6f0062e936
paper_authors.avatarUrl: /avatars/3d4d68830fafd35f7d3769b58149ff34.svg
paper_authors.fullname: Dongjun Kim
paper_authors.type: user
paper_authors.user: partitionsofunity
paper_authors._id: 66588e3daad054f64123df05
paper_authors.name: Takashi Shibuya
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:14:49.051Z
paper_authors._id: 6650773ca6acfdd2aba7d486
paper_authors.avatarUrl: /avatars/d297886ea60dbff98a043caf825820ed.svg
paper_authors.fullname: Takashi Shibuya
paper_authors.type: user
paper_authors.user: TakashiShibuyaSony
paper_authors._id: 66588e3daad054f64123df06
paper_authors.name: Chieh-Hsin Lai
paper_authors._id: 66588e3daad054f64123df07
paper_authors.name: Zhi Zhong
paper_authors._id: 66588e3daad054f64123df08
paper_authors.name: Yuhta Takida
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:15:08.191Z
paper_authors._id: 66138c4074f830bc7d9d6622
paper_authors.avatarUrl: /avatars/d50da6d7597d3bcf63f9f0c74e910155.svg
paper_authors.fullname: Yuhta Takida
paper_authors.type: user
paper_authors.user: ytakida
paper_authors._id: 66588e3daad054f64123df09
paper_authors.name: Yuki Mitsufuji
paper_title: SoundCTM: Uniting Score-based and Consistency Models for Text-to-Sound
    Generation
paper_summary: Sound content is an indispensable element for multimedia works such as video
  games, music, and films. Recent high-quality diffusion-based sound generation
  models can serve as valuable tools for the creators. However, despite producing
  high-quality sounds, these models often suffer from slow inference speeds. This
  drawback burdens creators, who typically refine their sounds through trial and
  error to align them with their artistic intentions. To address this issue, we
  introduce Sound Consistency Trajectory Models (SoundCTM). Our model enables
  flexible transitioning between high-quality 1-step sound generation and
  superior sound quality through multi-step generation. This allows creators to
  initially control sounds with 1-step samples before refining them through
  multi-step generation. While CTM fundamentally achieves flexible 1-step and
  multi-step generation, its impressive performance heavily depends on an
  additional pretrained feature extractor and an adversarial loss, which are
  expensive to train and not always available in other domains. Thus, we reframe
  CTM's training framework and introduce a novel feature distance by utilizing
  the teacher's network for a distillation loss. Additionally, while distilling
  classifier-free guided trajectories, we train conditional and unconditional
  student models simultaneously and interpolate between these models during
  inference. We also propose training-free controllable frameworks for SoundCTM,
  leveraging its flexible sampling capability. SoundCTM achieves both promising
  1-step and multi-step real-time sound generation without using any extra
  off-the-shelf networks. Furthermore, we demonstrate SoundCTM's capability of
  controllable sound generation in a training-free manner.
paper_ai_summary: SoundCTM efficiently generates high-quality sounds in real-time using both 1-step and multi-step methods, employing a teacher-student framework and controllable sound generation without additional networks.
paper_ai_keywords: diffusion-based sound generation Sound Consistency Trajectory Models (SoundCTM) 1-step sound generation multi-step generation pretrained feature extractor adversarial loss distillation loss conditional models unconditional models feature distance classifier-free guided trajectories training-free controllable frameworks