JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability
  and Aesthetic Alignment
Diffusion and flow-matching models have revolutionized automatic
text-to-audio generation in recent times. These models are increasingly capable
of generating high quality and faithful audio outputs capturing to speech and
acoustic events. However, there is still much room for improvement in creative
audio generation that primarily involves music and songs. Recent open
lyrics-to-song models, such as, DiffRhythm, ACE-Step, and LeVo, have set an
acceptable standard in automatic song generation for recreational use. However,
these models lack fine-grained word-level controllability often desired by
musicians in their workflows. To the best of our knowledge, our
flow-matching-based JAM is the first effort toward endowing word-level timing
and duration control in song generation, allowing fine-grained vocal control.
To enhance the quality of generated songs to better align with human
preferences, we implement aesthetic alignment through Direct Preference
Optimization, which iteratively refines the model using a synthetic dataset,
eliminating the need or manual data annotations. Furthermore, we aim to
standardize the evaluation of such lyrics-to-song models through our public
evaluation dataset JAME. We show that JAM outperforms the existing models in
terms of the music-specific attributes.
688826c8af872d625c10c5a1
Renhang Liu
688826c8af872d625c10c5a2
Chia-Yu Hung
claimed_verified
2025-07-29T07:08:59.724Z
62864cdf011a13fe5f84d746
https://cdn-avatars.huggingface.co/v1/production/uploads/62864cdf011a13fe5f84d746/5yPYgAtGJTaUq4fHhh9Pr.jpeg
Hung Chia Yu
user
hungchiayu
688826c8af872d625c10c5a3
Navonil Majumder
688826c8af872d625c10c5a4
Taylor Gautreaux
688826c8af872d625c10c5a5
Amir Ali Bagherzadeh
688826c8af872d625c10c5a6
Chuan Li
688826c8af872d625c10c5a7
Dorien Herremans
688826c8af872d625c10c5a8
Soujanya Poria
JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability
  and Aesthetic Alignment
Diffusion and flow-matching models have revolutionized automatic
text-to-audio generation in recent times. These models are increasingly capable
of generating high quality and faithful audio outputs capturing to speech and
acoustic events. However, there is still much room for improvement in creative
audio generation that primarily involves music and songs. Recent open
lyrics-to-song models, such as, DiffRhythm, ACE-Step, and LeVo, have set an
acceptable standard in automatic song generation for recreational use. However,
these models lack fine-grained word-level controllability often desired by
musicians in their workflows. To the best of our knowledge, our
flow-matching-based JAM is the first effort toward endowing word-level timing
and duration control in song generation, allowing fine-grained vocal control.
To enhance the quality of generated songs to better align with human
preferences, we implement aesthetic alignment through Direct Preference
Optimization, which iteratively refines the model using a synthetic dataset,
eliminating the need or manual data annotations. Furthermore, we aim to
standardize the evaluation of such lyrics-to-song models through our public
evaluation dataset JAME. We show that JAM outperforms the existing models in
terms of the music-specific attributes.
JAM, a flow-matching-based model, introduces word-level timing and duration control in song generation and uses Direct Preference Optimization to enhance audio quality, outperforming existing models in music-specific attributes.
diffusion models flow-matching models text-to-audio generation automatic song generation lyrics-to-song models DiffRhythm ACE-Step LeVo word-level controllability vocal control aesthetic alignment Direct Preference Optimization synthetic dataset public evaluation dataset JAME