title: DITTO: Diffusion Inference-Time T-Optimization for Music Generation
summary: We propose Diffusion Inference-Time T-Optimization (DITTO), a general-purpose
  frame-work for controlling pre-trained text-to-music diffusion models at
  inference-time via optimizing initial noise latents. Our method can be used to
  optimize through any differentiable feature matching loss to achieve a target
  (stylized) output and leverages gradient checkpointing for memory efficiency.
  We demonstrate a surprisingly wide-range of applications for music generation
  including inpainting, outpainting, and looping as well as intensity, melody,
  and musical structure control - all without ever fine-tuning the underlying
  model. When we compare our approach against related training, guidance, and
  optimization-based methods, we find DITTO achieves state-of-the-art performance
  on nearly all tasks, including outperforming comparable approaches on
  controllability, audio quality, and computational efficiency, thus opening the
  door for high-quality, flexible, training-free control of diffusion models.
  Sound examples can be found at https://DITTO-Music.github.io/web/.
paper_authors._id: 65af528552e1b2aae42443fb
paper_authors.name: Zachary Novack
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-01-23T05:47:52.680Z
paper_authors._id: 643060c6cb3fe707b24c53a2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643060c6cb3fe707b24c53a2/MIoM9hrX0vV4XRyrm-4Kz.jpeg
paper_authors.fullname: Zachary Novack
paper_authors.type: user
paper_authors.user: ZacharyNovack
paper_authors._id: 65af528552e1b2aae42443fc
paper_authors.name: Julian McAuley
paper_authors._id: 65af528552e1b2aae42443fd
paper_authors.name: Taylor Berg-Kirkpatrick
paper_authors._id: 65af528552e1b2aae42443fe
paper_authors.name: Nicholas J. Bryan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T06:35:21.161Z
paper_authors._id: 648119ee7a741c7f33f49f25
paper_authors.avatarUrl: /avatars/183d2143ba20e1cc8712c63c055aadd7.svg
paper_authors.fullname: Nicholas J. Bryan
paper_authors.type: user
paper_authors.user: Njb
paper_title: DITTO: Diffusion Inference-Time T-Optimization for Music Generation
paper_summary: We propose Diffusion Inference-Time T-Optimization (DITTO), a general-purpose
  frame-work for controlling pre-trained text-to-music diffusion models at
  inference-time via optimizing initial noise latents. Our method can be used to
  optimize through any differentiable feature matching loss to achieve a target
  (stylized) output and leverages gradient checkpointing for memory efficiency.
  We demonstrate a surprisingly wide-range of applications for music generation
  including inpainting, outpainting, and looping as well as intensity, melody,
  and musical structure control - all without ever fine-tuning the underlying
  model. When we compare our approach against related training, guidance, and
  optimization-based methods, we find DITTO achieves state-of-the-art performance
  on nearly all tasks, including outperforming comparable approaches on
  controllability, audio quality, and computational efficiency, thus opening the
  door for high-quality, flexible, training-free control of diffusion models.
  Sound examples can be found at https://DITTO-Music.github.io/web/.
paper_ai_summary: DITTO optimizes noise latents during inference to control text-to-music diffusion models, achieving superior performance in various tasks without fine-tuning.
paper_ai_keywords: Diffusion Inference-Time T-Optimization pre-trained text-to-music diffusion models initial noise latents differentiable feature matching loss gradient checkpointing music generation inpainting outpainting looping intensity control melody control musical structure control training-free control