MuseControlLite: Multifunctional Music Generation with Lightweight
  Conditioners
We propose MuseControlLite, a lightweight mechanism designed to fine-tune
text-to-music generation models for precise conditioning using various
time-varying musical attributes and reference audio signals. The key finding is
that positional embeddings, which have been seldom used by text-to-music
generation models in the conditioner for text conditions, are critical when the
condition of interest is a function of time. Using melody control as an
example, our experiments show that simply adding rotary positional embeddings
to the decoupled cross-attention layers increases control accuracy from 56.6%
to 61.1%, while requiring 6.75 times fewer trainable parameters than
state-of-the-art fine-tuning mechanisms, using the same pre-trained diffusion
Transformer model of Stable Audio Open. We evaluate various forms of musical
attribute control, audio inpainting, and audio outpainting, demonstrating
improved controllability over MusicGen-Large and Stable Audio Open ControlNet
at a significantly lower fine-tuning cost, with only 85M trainble parameters.
Source code, model checkpoints, and demo examples are available at:
https://musecontrollite.github.io/web/.
685d3bfd696820ba1f28f3b7
Fang-Duo Tsai
claimed_verified
2025-06-27T08:57:21.464Z
6665b1f48c8082c85956a038
/avatars/bed84c343deb4c10e8165a501b152e79.svg
Fang Duo Tsai
user
fundwotsai2001
685d3bfd696820ba1f28f3b8
Shih-Lun Wu
685d3bfd696820ba1f28f3b9
Weijaw Lee
685d3bfd696820ba1f28f3ba
Sheng-Ping Yang
685d3bfd696820ba1f28f3bb
Bo-Rui Chen
685d3bfd696820ba1f28f3bc
Hao-Chung Cheng
685d3bfd696820ba1f28f3bd
Yi-Hsuan Yang
MuseControlLite: Multifunctional Music Generation with Lightweight
  Conditioners
We propose MuseControlLite, a lightweight mechanism designed to fine-tune
text-to-music generation models for precise conditioning using various
time-varying musical attributes and reference audio signals. The key finding is
that positional embeddings, which have been seldom used by text-to-music
generation models in the conditioner for text conditions, are critical when the
condition of interest is a function of time. Using melody control as an
example, our experiments show that simply adding rotary positional embeddings
to the decoupled cross-attention layers increases control accuracy from 56.6%
to 61.1%, while requiring 6.75 times fewer trainable parameters than
state-of-the-art fine-tuning mechanisms, using the same pre-trained diffusion
Transformer model of Stable Audio Open. We evaluate various forms of musical
attribute control, audio inpainting, and audio outpainting, demonstrating
improved controllability over MusicGen-Large and Stable Audio Open ControlNet
at a significantly lower fine-tuning cost, with only 85M trainble parameters.
Source code, model checkpoints, and demo examples are available at:
https://musecontrollite.github.io/web/.
Rotary positional embeddings enhance time-varying control in text-to-music generation models with fewer parameters.
positional embeddings rotary positional embeddings cross-attention layers decoupled cross-attention layers diffusion Transformer MusicGen-Large Stable Audio Open ControlNet audio inpainting audio outpainting