title: Efficient Neural Music Generation
summary: Recent progress in music generation has been remarkably advanced by the
  state-of-the-art MusicLM, which comprises a hierarchy of three LMs,
  respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet,
  sampling with the MusicLM requires processing through these LMs one by one to
  obtain the fine-grained acoustic tokens, making it computationally expensive
  and prohibitive for a real-time generation. Efficient music generation with a
  quality on par with MusicLM remains a significant challenge. In this paper, we
  present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion
  model that generates music audios of state-of-the-art quality meanwhile
  reducing 95.7% or 99.6% forward passes in MusicLM, respectively, for sampling
  10s or 30s music. MeLoDy inherits the highest-level LM from MusicLM for
  semantic modeling, and applies a novel dual-path diffusion (DPD) model and an
  audio VAE-GAN to efficiently decode the conditioning semantic tokens into
  waveform. DPD is proposed to simultaneously model the coarse and fine acoustics
  by incorporating the semantic information into segments of latents effectively
  via cross-attention at each denoising step. Our experimental results suggest
  the superiority of MeLoDy, not only in its practical advantages on sampling
  speed and infinitely continuable generation, but also in its state-of-the-art
  musicality, audio quality, and text correlation.
    Our samples are available at https://Efficient-MeLoDy.github.io/.
paper_authors._id: 64701313036229a8a71cf8b0
paper_authors.name: Max W. Y. Lam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T13:56:34.928Z
paper_authors._id: 6278d25ae8e956767ec7e6bd
paper_authors.avatarUrl: /avatars/59ce8fdee94f113b3334eea600a34ca9.svg
paper_authors.fullname: Max W. Y. Lam
paper_authors.type: user
paper_authors.user: maxingaussian
paper_authors._id: 64701313036229a8a71cf8b1
paper_authors.name: Qiao Tian
paper_authors._id: 64701313036229a8a71cf8b2
paper_authors.name: Tang Li
paper_authors._id: 64701313036229a8a71cf8b3
paper_authors.name: Zongyu Yin
paper_authors._id: 64701313036229a8a71cf8b4
paper_authors.name: Siyuan Feng
paper_authors._id: 64701313036229a8a71cf8b5
paper_authors.name: Ming Tu
paper_authors._id: 64701313036229a8a71cf8b6
paper_authors.name: Yuliang Ji
paper_authors._id: 64701313036229a8a71cf8b7
paper_authors.name: Rui Xia
paper_authors._id: 64701313036229a8a71cf8b8
paper_authors.name: Mingbo Ma
paper_authors._id: 64701313036229a8a71cf8b9
paper_authors.name: Xuchen Song
paper_authors._id: 64701313036229a8a71cf8ba
paper_authors.name: Jitong Chen
paper_authors._id: 64701313036229a8a71cf8bb
paper_authors.name: Yuping Wang
paper_authors._id: 64701313036229a8a71cf8bc
paper_authors.name: Yuxuan Wang
paper_title: Efficient Neural Music Generation
paper_summary: Recent progress in music generation has been remarkably advanced by the
  state-of-the-art MusicLM, which comprises a hierarchy of three LMs,
  respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet,
  sampling with the MusicLM requires processing through these LMs one by one to
  obtain the fine-grained acoustic tokens, making it computationally expensive
  and prohibitive for a real-time generation. Efficient music generation with a
  quality on par with MusicLM remains a significant challenge. In this paper, we
  present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion
  model that generates music audios of state-of-the-art quality meanwhile
  reducing 95.7% or 99.6% forward passes in MusicLM, respectively, for sampling
  10s or 30s music. MeLoDy inherits the highest-level LM from MusicLM for
  semantic modeling, and applies a novel dual-path diffusion (DPD) model and an
  audio VAE-GAN to efficiently decode the conditioning semantic tokens into
  waveform. DPD is proposed to simultaneously model the coarse and fine acoustics
  by incorporating the semantic information into segments of latents effectively
  via cross-attention at each denoising step. Our experimental results suggest
  the superiority of MeLoDy, not only in its practical advantages on sampling
  speed and infinitely continuable generation, but also in its state-of-the-art
  musicality, audio quality, and text correlation.
    Our samples are available at https://Efficient-MeLoDy.github.io/.
paper_ai_keywords: MusicLM semantic modeling coarse acoustic modeling fine acoustic modeling LM-guided diffusion model dual-path diffusion (DPD) audio VAE-GAN cross-differentiation denoising steps infinitely continuable generation