Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think
<|reference_start|>Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think: Recent work showed that large diffusion models can be reused as highly precise monocular depth estimators by casting depth estimation as an image-conditional image generation task. While the proposed model achieved state-of-the-art results, high computational demands due to multi-step inference limited its use in many scenarios. In this paper, we show that the perceived inefficiency was caused by a flaw in the inference pipeline that has so far gone unnoticed. The fixed model performs comparably to the best previously reported configuration while being more than 200$\times$ faster. To optimize for downstream task performance, we perform end-to-end fine-tuning on top of the single-step model with task-specific losses and get a deterministic model that outperforms all other diffusion-based depth and normal estimation models on common zero-shot benchmarks. We surprisingly find that this fine-tuning protocol also works directly on Stable Diffusion and achieves comparable performance to current state-of-the-art diffusion-based depth and normal estimation models, calling into question some of the conclusions drawn from prior works.<|reference_end|>
@article{garcia2024fine-tuning,
    title={Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think},
    author={Gonzalo Martin Garcia, Karim Abou Zeid, Christian Schmidt, Daan de
  Geus, Alexander Hermans, Bastian Leibe},
    journal={arXiv preprint arXiv:2409.11355},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.11355},
    primaryClass={cs.CV}
}