title: Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think
summary: Recent work showed that large diffusion models can be reused as highly
  precise monocular depth estimators by casting depth estimation as an
  image-conditional image generation task. While the proposed model achieved
  state-of-the-art results, high computational demands due to multi-step
  inference limited its use in many scenarios. In this paper, we show that the
  perceived inefficiency was caused by a flaw in the inference pipeline that has
  so far gone unnoticed. The fixed model performs comparably to the best
  previously reported configuration while being more than 200times faster. To
  optimize for downstream task performance, we perform end-to-end fine-tuning on
  top of the single-step model with task-specific losses and get a deterministic
  model that outperforms all other diffusion-based depth and normal estimation
  models on common zero-shot benchmarks. We surprisingly find that this
  fine-tuning protocol also works directly on Stable Diffusion and achieves
  comparable performance to current state-of-the-art diffusion-based depth and
  normal estimation models, calling into question some of the conclusions drawn
  from prior works.
paper_authors._id: 66ea40c200ac2066be49bf21
paper_authors.name: Gonzalo Martin Garcia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T08:02:39.128Z
paper_authors._id: 65c78db0de6bfaeded594924
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c78db0de6bfaeded594924/Nngmla4CS7twp54uCOcA-.png
paper_authors.fullname: Gonzalo Martin Garcia
paper_authors.type: user
paper_authors.user: GonzaloMG
paper_authors._id: 66ea40c200ac2066be49bf22
paper_authors.name: Karim Abou Zeid
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-18T07:01:25.614Z
paper_authors._id: 63d50ffa695c73ecef30cc34
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63d50ffa695c73ecef30cc34/ocahS59a6EDsQSM6uHZ5l.jpeg
paper_authors.fullname: Karim Abou Zeid
paper_authors.type: user
paper_authors.user: kabouzeid
paper_authors._id: 66ea40c200ac2066be49bf23
paper_authors.name: Christian Schmidt
paper_authors._id: 66ea40c200ac2066be49bf24
paper_authors.name: Daan de Geus
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T08:02:49.437Z
paper_authors._id: 6392408ae836c1426b776472
paper_authors.avatarUrl: /avatars/380b66f142197572e4d8516046c82508.svg
paper_authors.fullname: Daan de Geus
paper_authors.type: user
paper_authors.user: ddegeus
paper_authors._id: 66ea40c200ac2066be49bf25
paper_authors.name: Alexander Hermans
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-19T07:07:00.765Z
paper_authors._id: 631b65d65b628ccf3ce5fef6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631b65d65b628ccf3ce5fef6/7zFpgePJX5gOlRqPRO6sr.png
paper_authors.fullname: Alexander Hermans
paper_authors.type: user
paper_authors.user: Pandoro
paper_authors._id: 66ea40c200ac2066be49bf26
paper_authors.name: Bastian Leibe
paper_title: Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think
paper_summary: Recent work showed that large diffusion models can be reused as highly
  precise monocular depth estimators by casting depth estimation as an
  image-conditional image generation task. While the proposed model achieved
  state-of-the-art results, high computational demands due to multi-step
  inference limited its use in many scenarios. In this paper, we show that the
  perceived inefficiency was caused by a flaw in the inference pipeline that has
  so far gone unnoticed. The fixed model performs comparably to the best
  previously reported configuration while being more than 200times faster. To
  optimize for downstream task performance, we perform end-to-end fine-tuning on
  top of the single-step model with task-specific losses and get a deterministic
  model that outperforms all other diffusion-based depth and normal estimation
  models on common zero-shot benchmarks. We surprisingly find that this
  fine-tuning protocol also works directly on Stable Diffusion and achieves
  comparable performance to current state-of-the-art diffusion-based depth and
  normal estimation models, calling into question some of the conclusions drawn
  from prior works.
paper_ai_summary: Efficient depth estimation is achieved by optimizing the inference pipeline of diffusion models and fine-tuning them for specific tasks, outperforming existing methods.
paper_ai_keywords: diffusion models image-conditional image generation monocular depth estimation inference pipeline end-to-end fine-tuning task-specific losses zero-shot benchmarks Stable Diffusion