DEFT: Efficient Finetuning of Conditional Diffusion Models by Learning the Generalised $h$-transform
<|reference_start|>DEFT: Efficient Finetuning of Conditional Diffusion Models by Learning the Generalised $h$-transform: Generative modelling paradigms based on denoising diffusion processes have emerged as a leading candidate for conditional sampling in inverse problems. In many real-world applications, we often have access to large, expensively trained unconditional diffusion models, which we aim to exploit for improving conditional sampling. Most recent approaches are motivated heuristically and lack a unifying framework, obscuring connections between them. Further, they often suffer from issues such as being very sensitive to hyperparameters, being expensive to train or needing access to weights hidden behind a closed API. In this work, we unify conditional training and sampling using the mathematically well-understood Doob's h-transform. This new perspective allows us to unify many existing methods under a common umbrella. Under this framework, we propose DEFT (Doob's h-transform Efficient FineTuning), a new approach for conditional generation that simply fine-tunes a very small network to quickly learn the conditional $h$-transform, while keeping the larger unconditional network unchanged. DEFT is much faster than existing baselines while achieving state-of-the-art performance across a variety of linear and non-linear benchmarks. On image reconstruction tasks, we achieve speedups of up to 1.6$\times$, while having the best perceptual quality on natural images and reconstruction performance on medical images.<|reference_end|>
@article{denker2024deft:,
    title={DEFT: Efficient Fine-Tuning of Diffusion Models by Learning the
  Generalised $h$-transform},
    author={Alexander Denker, Francisco Vargas, Shreyas Padhy, Kieran Didi, Simon
  Mathis, Vincent Dutordoir, Riccardo Barbano, Emile Mathieu, Urszula Julia
  Komorowska, Pietro Lio},
    journal={arXiv preprint arXiv:2406.01781},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01781},
    primaryClass={cs.LG}
}