Efficient Neural Music Generation
Recent progress in music generation has been remarkably advanced by the
state-of-the-art MusicLM, which comprises a hierarchy of three LMs,
respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet,
sampling with the MusicLM requires processing through these LMs one by one to
obtain the fine-grained acoustic tokens, making it computationally expensive
and prohibitive for a real-time generation. Efficient music generation with a
quality on par with MusicLM remains a significant challenge. In this paper, we
present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion
model that generates music audios of state-of-the-art quality meanwhile
reducing 95.7% or 99.6% forward passes in MusicLM, respectively, for sampling
10s or 30s music. MeLoDy inherits the highest-level LM from MusicLM for
semantic modeling, and applies a novel dual-path diffusion (DPD) model and an
audio VAE-GAN to efficiently decode the conditioning semantic tokens into
waveform. DPD is proposed to simultaneously model the coarse and fine acoustics
by incorporating the semantic information into segments of latents effectively
via cross-attention at each denoising step. Our experimental results suggest
the superiority of MeLoDy, not only in its practical advantages on sampling
speed and infinitely continuable generation, but also in its state-of-the-art
musicality, audio quality, and text correlation.
  Our samples are available at https://Efficient-MeLoDy.github.io/.
64701313036229a8a71cf8b0
Max W. Y. Lam
admin_assigned
2023-05-26T13:56:34.928Z
6278d25ae8e956767ec7e6bd
/avatars/59ce8fdee94f113b3334eea600a34ca9.svg
Max W. Y. Lam
user
maxingaussian
64701313036229a8a71cf8b1
Qiao Tian
64701313036229a8a71cf8b2
Tang Li
64701313036229a8a71cf8b3
Zongyu Yin
64701313036229a8a71cf8b4
Siyuan Feng
64701313036229a8a71cf8b5
Ming Tu
64701313036229a8a71cf8b6
Yuliang Ji
64701313036229a8a71cf8b7
Rui Xia
64701313036229a8a71cf8b8
Mingbo Ma
64701313036229a8a71cf8b9
Xuchen Song
claimed_verified
2025-03-28T08:39:48.616Z
6462b241b438438da3c25a5d
/avatars/606a67f1be639c9a5e36f293abd5f27a.svg
Xuchen Song
user
xuchensong
64701313036229a8a71cf8ba
Jitong Chen
64701313036229a8a71cf8bb
Yuping Wang
64701313036229a8a71cf8bc
Yuxuan Wang
Efficient Neural Music Generation
Recent progress in music generation has been remarkably advanced by the
state-of-the-art MusicLM, which comprises a hierarchy of three LMs,
respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet,
sampling with the MusicLM requires processing through these LMs one by one to
obtain the fine-grained acoustic tokens, making it computationally expensive
and prohibitive for a real-time generation. Efficient music generation with a
quality on par with MusicLM remains a significant challenge. In this paper, we
present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion
model that generates music audios of state-of-the-art quality meanwhile
reducing 95.7% or 99.6% forward passes in MusicLM, respectively, for sampling
10s or 30s music. MeLoDy inherits the highest-level LM from MusicLM for
semantic modeling, and applies a novel dual-path diffusion (DPD) model and an
audio VAE-GAN to efficiently decode the conditioning semantic tokens into
waveform. DPD is proposed to simultaneously model the coarse and fine acoustics
by incorporating the semantic information into segments of latents effectively
via cross-attention at each denoising step. Our experimental results suggest
the superiority of MeLoDy, not only in its practical advantages on sampling
speed and infinitely continuable generation, but also in its state-of-the-art
musicality, audio quality, and text correlation.
  Our samples are available at https://Efficient-MeLoDy.github.io/.
MeLoDy is an LM-guided diffusion model that generates high-quality music with reduced computational cost and improved sampling speed compared to MusicLM.
MusicLM semantic modeling dual-path diffusion audio VAE-GAN cross-attention coarse acoustics fine acoustics musicality audio quality text correlation