DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music
  Generation
Controllable music generation methods are critical for human-centered
AI-based music creation, but are currently limited by speed, quality, and
control design trade-offs. Diffusion Inference-Time T-optimization (DITTO), in
particular, offers state-of-the-art results, but is over 10x slower than
real-time, limiting practical use. We propose Distilled Diffusion
Inference-Time T -Optimization (or DITTO-2), a new method to speed up
inference-time optimization-based control and unlock faster-than-real-time
generation for a wide-variety of applications such as music inpainting,
outpainting, intensity, melody, and musical structure control. Our method works
by (1) distilling a pre-trained diffusion model for fast sampling via an
efficient, modified consistency or consistency trajectory distillation process
(2) performing inference-time optimization using our distilled model with
one-step sampling as an efficient surrogate optimization task and (3) running a
final multi-step sampling generation (decoding) using our estimated noise
latents for best-quality, fast, controllable generation. Through thorough
evaluation, we find our method not only speeds up generation over 10-20x, but
simultaneously improves control adherence and generation quality all at once.
Furthermore, we apply our approach to a new application of maximizing text
adherence (CLAP score) and show we can convert an unconditional diffusion model
without text inputs into a model that yields state-of-the-art text control.
Sound examples can be found at https://ditto-music.github.io/ditto2/.
665946a11cc36f01008ce18b
Zachary Novack
extracted_confirmed
2024-05-31T03:45:43.811Z
643060c6cb3fe707b24c53a2
https://cdn-avatars.huggingface.co/v1/production/uploads/643060c6cb3fe707b24c53a2/MIoM9hrX0vV4XRyrm-4Kz.jpeg
Zachary Novack
user
ZacharyNovack
665946a11cc36f01008ce18c
Julian McAuley
665946a11cc36f01008ce18d
Taylor Berg-Kirkpatrick
665946a11cc36f01008ce18e
Nicholas Bryan
claimed_verified
2024-10-08T07:22:43.201Z
648119ee7a741c7f33f49f25
/avatars/183d2143ba20e1cc8712c63c055aadd7.svg
Nicholas J. Bryan
user
Njb
DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music
  Generation
Controllable music generation methods are critical for human-centered
AI-based music creation, but are currently limited by speed, quality, and
control design trade-offs. Diffusion Inference-Time T-optimization (DITTO), in
particular, offers state-of-the-art results, but is over 10x slower than
real-time, limiting practical use. We propose Distilled Diffusion
Inference-Time T -Optimization (or DITTO-2), a new method to speed up
inference-time optimization-based control and unlock faster-than-real-time
generation for a wide-variety of applications such as music inpainting,
outpainting, intensity, melody, and musical structure control. Our method works
by (1) distilling a pre-trained diffusion model for fast sampling via an
efficient, modified consistency or consistency trajectory distillation process
(2) performing inference-time optimization using our distilled model with
one-step sampling as an efficient surrogate optimization task and (3) running a
final multi-step sampling generation (decoding) using our estimated noise
latents for best-quality, fast, controllable generation. Through thorough
evaluation, we find our method not only speeds up generation over 10-20x, but
simultaneously improves control adherence and generation quality all at once.
Furthermore, we apply our approach to a new application of maximizing text
adherence (CLAP score) and show we can convert an unconditional diffusion model
without text inputs into a model that yields state-of-the-art text control.
Sound examples can be found at https://ditto-music.github.io/ditto2/.
A new method, DITTO-2, accelerates diffusion model inference for real-time music generation while enhancing control and quality, including text adherence.
diffusion models inference-time optimization music inpainting outpainting intensity melody musical structure control consistency trajectory distillation one-step sampling multi-step sampling CLAP score unconditional diffusion model