Tango 2: Aligning Diffusion-based Text-to-Audio Generations through
  Direct Preference Optimization
Generative multimodal content is increasingly prevalent in much of the
content creation arena, as it has the potential to allow artists and media
personnel to create pre-production mockups by quickly bringing their ideas to
life. The generation of audio from text prompts is an important aspect of such
processes in the music and film industry. Many of the recent diffusion-based
text-to-audio models focus on training increasingly sophisticated diffusion
models on a large set of datasets of prompt-audio pairs. These models do not
explicitly focus on the presence of concepts or events and their temporal
ordering in the output audio with respect to the input prompt. Our hypothesis
is focusing on how these aspects of audio generation could improve audio
generation performance in the presence of limited data. As such, in this work,
using an existing text-to-audio model Tango, we synthetically create a
preference dataset where each prompt has a winner audio output and some loser
audio outputs for the diffusion model to learn from. The loser outputs, in
theory, have some concepts from the prompt missing or in an incorrect order. We
fine-tune the publicly available Tango text-to-audio model using diffusion-DPO
(direct preference optimization) loss on our preference dataset and show that
it leads to improved audio output over Tango and AudioLDM2, in terms of both
automatic- and manual-evaluation metrics.
661de5560a9d678b6f653cea
Navonil Majumder
admin_assigned
2024-04-18T10:07:28.728Z
644e20f267c9458c912bd246
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5j7LZ-6sCarsEvKr2tPAj.png
Navonil Majumder
user
hollownight
661de5560a9d678b6f653ceb
Chia-Yu Hung
claimed_verified
2024-06-02T18:48:05.513Z
62864cdf011a13fe5f84d746
https://cdn-avatars.huggingface.co/v1/production/uploads/62864cdf011a13fe5f84d746/5yPYgAtGJTaUq4fHhh9Pr.jpeg
Hung Chia Yu
user
hungchiayu
661de5560a9d678b6f653cec
Deepanway Ghosal
claimed_verified
2024-04-16T23:16:51.115Z
631246a8f568fb0098f5e72e
https://cdn-avatars.huggingface.co/v1/production/uploads/631246a8f568fb0098f5e72e/kTcs1cgwFqd1Wjr3tolcL.jpeg
Deepanway
user
deepanway
661de5560a9d678b6f653ced
Wei-Ning Hsu
admin_assigned
2024-04-18T10:07:41.126Z
64c143d484887aa2b43bac0d
/avatars/817dba24739676aa0f34692702bb00d6.svg
Wei-Ning Hsu
user
wnhsu
661de5560a9d678b6f653cee
Rada Mihalcea
admin_assigned
2024-04-18T10:07:46.862Z
64153d8f95fb6f824b22d960
/avatars/e189e792dbdb0e2a8a04eda3a8c2bcfc.svg
Rada Mihalcea
user
mihalcea
661de5560a9d678b6f653cef
Soujanya Poria
claimed_verified
2024-04-16T12:44:26.953Z
626b626405fe1cb65725aca1
https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/aa-Lata46I3fXOmMetvXH.jpeg
Soujanya Poria
user
soujanyaporia
Tango 2: Aligning Diffusion-based Text-to-Audio Generations through
  Direct Preference Optimization
Generative multimodal content is increasingly prevalent in much of the
content creation arena, as it has the potential to allow artists and media
personnel to create pre-production mockups by quickly bringing their ideas to
life. The generation of audio from text prompts is an important aspect of such
processes in the music and film industry. Many of the recent diffusion-based
text-to-audio models focus on training increasingly sophisticated diffusion
models on a large set of datasets of prompt-audio pairs. These models do not
explicitly focus on the presence of concepts or events and their temporal
ordering in the output audio with respect to the input prompt. Our hypothesis
is focusing on how these aspects of audio generation could improve audio
generation performance in the presence of limited data. As such, in this work,
using an existing text-to-audio model Tango, we synthetically create a
preference dataset where each prompt has a winner audio output and some loser
audio outputs for the diffusion model to learn from. The loser outputs, in
theory, have some concepts from the prompt missing or in an incorrect order. We
fine-tune the publicly available Tango text-to-audio model using diffusion-DPO
(direct preference optimization) loss on our preference dataset and show that
it leads to improved audio output over Tango and AudioLDM2, in terms of both
automatic- and manual-evaluation metrics.
Improving text-to-audio generation by fine-tuning the Tango model using diffusion-DPO on a preference dataset enhances audio quality in terms of automatic and manual evaluations.
diffusion models preference dataset text-to-audio models Tango diffusion-DPO direct preference optimization audio generation automatic-evaluation metrics manual-evaluation metrics AudioLDM2