title: DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music
    Generation
summary: Controllable music generation methods are critical for human-centered
  AI-based music creation, but are currently limited by speed, quality, and
  control design trade-offs. Diffusion Inference-Time T-optimization (DITTO), in
  particular, offers state-of-the-art results, but is over 10x slower than
  real-time, limiting practical use. We propose Distilled Diffusion
  Inference-Time T -Optimization (or DITTO-2), a new method to speed up
  inference-time optimization-based control and unlock faster-than-real-time
  generation for a wide-variety of applications such as music inpainting,
  outpainting, intensity, melody, and musical structure control. Our method works
  by (1) distilling a pre-trained diffusion model for fast sampling via an
  efficient, modified consistency or consistency trajectory distillation process
  (2) performing inference-time optimization using our distilled model with
  one-step sampling as an efficient surrogate optimization task and (3) running a
  final multi-step sampling generation (decoding) using our estimated noise
  latents for best-quality, fast, controllable generation. Through thorough
  evaluation, we find our method not only speeds up generation over 10-20x, but
  simultaneously improves control adherence and generation quality all at once.
  Furthermore, we apply our approach to a new application of maximizing text
  adherence (CLAP score) and show we can convert an unconditional diffusion model
  without text inputs into a model that yields state-of-the-art text control.
  Sound examples can be found at https://ditto-music.github.io/ditto2/.
paper_authors._id: 665946a11cc36f01008ce18b
paper_authors.name: Zachary Novack
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-31T03:45:43.811Z
paper_authors._id: 643060c6cb3fe707b24c53a2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643060c6cb3fe707b24c53a2/MIoM9hrX0vV4XRyrm-4Kz.jpeg
paper_authors.fullname: Zachary Novack
paper_authors.type: user
paper_authors.user: ZacharyNovack
paper_authors._id: 665946a11cc36f01008ce18c
paper_authors.name: Julian McAuley
paper_authors._id: 665946a11cc36f01008ce18d
paper_authors.name: Taylor Berg-Kirkpatrick
paper_authors._id: 665946a11cc36f01008ce18e
paper_authors.name: Nicholas Bryan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:22:43.201Z
paper_authors._id: 648119ee7a741c7f33f49f25
paper_authors.avatarUrl: /avatars/183d2143ba20e1cc8712c63c055aadd7.svg
paper_authors.fullname: Nicholas J. Bryan
paper_authors.type: user
paper_authors.user: Njb
paper_title: DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music
    Generation
paper_summary: Controllable music generation methods are critical for human-centered
  AI-based music creation, but are currently limited by speed, quality, and
  control design trade-offs. Diffusion Inference-Time T-optimization (DITTO), in
  particular, offers state-of-the-art results, but is over 10x slower than
  real-time, limiting practical use. We propose Distilled Diffusion
  Inference-Time T -Optimization (or DITTO-2), a new method to speed up
  inference-time optimization-based control and unlock faster-than-real-time
  generation for a wide-variety of applications such as music inpainting,
  outpainting, intensity, melody, and musical structure control. Our method works
  by (1) distilling a pre-trained diffusion model for fast sampling via an
  efficient, modified consistency or consistency trajectory distillation process
  (2) performing inference-time optimization using our distilled model with
  one-step sampling as an efficient surrogate optimization task and (3) running a
  final multi-step sampling generation (decoding) using our estimated noise
  latents for best-quality, fast, controllable generation. Through thorough
  evaluation, we find our method not only speeds up generation over 10-20x, but
  simultaneously improves control adherence and generation quality all at once.
  Furthermore, we apply our approach to a new application of maximizing text
  adherence (CLAP score) and show we can convert an unconditional diffusion model
  without text inputs into a model that yields state-of-the-art text control.
  Sound examples can be found at https://ditto-music.github.io/ditto2/.
paper_ai_summary: A new method, DITTO-2, accelerates diffusion model inference for real-time music generation while enhancing control and quality, including text adherence.
paper_ai_keywords: diffusion models inference-time optimization music inpainting outpainting intensity melody musical structure control consistency trajectory distillation one-step sampling multi-step sampling CLAP score unconditional diffusion model