title: Marigold: Affordable Adaptation of Diffusion-Based Image Generators for
    Image Analysis
summary: The success of deep learning in computer vision over the past decade has
  hinged on large labeled datasets and strong pretrained models. In data-scarce
  settings, the quality of these pretrained models becomes crucial for effective
  transfer learning. Image classification and self-supervised learning have
  traditionally been the primary methods for pretraining CNNs and
  transformer-based architectures. Recently, the rise of text-to-image generative
  models, particularly those using denoising diffusion in a latent space, has
  introduced a new class of foundational models trained on massive, captioned
  image datasets. These models' ability to generate realistic images of unseen
  content suggests they possess a deep understanding of the visual world. In this
  work, we present Marigold, a family of conditional generative models and a
  fine-tuning protocol that extracts the knowledge from pretrained latent
  diffusion models like Stable Diffusion and adapts them for dense image analysis
  tasks, including monocular depth estimation, surface normals prediction, and
  intrinsic decomposition. Marigold requires minimal modification of the
  pre-trained latent diffusion model's architecture, trains with small synthetic
  datasets on a single GPU over a few days, and demonstrates state-of-the-art
  zero-shot generalization. Project page:
  https://marigoldcomputervision.github.io
paper_authors._id: 6825c0e6bfd4908da7747c41
paper_authors.name: Bingxin Ke
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-15T13:55:42.142Z
paper_authors._id: 63d90391da4f72339244c2a8
paper_authors.avatarUrl: /avatars/eb0e0259c391d59739c1a205c36bb539.svg
paper_authors.fullname: Bingxin Ke
paper_authors.type: user
paper_authors.user: Bingxin
paper_authors._id: 6825c0e6bfd4908da7747c42
paper_authors.name: Kevin Qu
paper_authors._id: 6825c0e6bfd4908da7747c43
paper_authors.name: Tianfu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-16T07:12:38.377Z
paper_authors._id: 633c25e000732349209d4e79
paper_authors.avatarUrl: /avatars/0fd853f6cbfcacc98035364585d58f2c.svg
paper_authors.fullname: Tianfu Wang
paper_authors.type: user
paper_authors.user: Tianfwang
paper_authors._id: 6825c0e6bfd4908da7747c44
paper_authors.name: Nando Metzger
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T13:45:09.533Z
paper_authors._id: 63a5785a8fb23d08bb2d0291
paper_authors.avatarUrl: /avatars/758b06dae06e9eee6fced10ce682aef1.svg
paper_authors.fullname: Nando Metzger
paper_authors.type: user
paper_authors.user: nandometzger
paper_authors._id: 6825c0e6bfd4908da7747c45
paper_authors.name: Shengyu Huang
paper_authors._id: 6825c0e6bfd4908da7747c46
paper_authors.name: Bo Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T09:03:44.866Z
paper_authors._id: 6591e8f6ac02633c0dcb2020
paper_authors.avatarUrl: /avatars/5a6e4db5bd3496948f94f68a849673f6.svg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: bobboli
paper_authors._id: 6825c0e6bfd4908da7747c47
paper_authors.name: Anton Obukhov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T13:45:11.907Z
paper_authors._id: 62f93abbc4817cfc0756b6f8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f93abbc4817cfc0756b6f8/rGYLaq-rmoJJYotkC1VXk.jpeg
paper_authors.fullname: Anton Obukhov
paper_authors.type: user
paper_authors.user: toshas
paper_authors._id: 6825c0e6bfd4908da7747c48
paper_authors.name: Konrad Schindler
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-15T10:53:52.126Z
paper_authors._id: 6750649b7edd6a98a1bbcd06
paper_authors.avatarUrl: /avatars/ba27f12d0333cf2d400d4405af7efe97.svg
paper_authors.fullname: Konrad Schindler
paper_authors.type: user
paper_authors.user: konradschindler
paper_title: Marigold: Affordable Adaptation of Diffusion-Based Image Generators for
    Image Analysis
paper_summary: The success of deep learning in computer vision over the past decade has
  hinged on large labeled datasets and strong pretrained models. In data-scarce
  settings, the quality of these pretrained models becomes crucial for effective
  transfer learning. Image classification and self-supervised learning have
  traditionally been the primary methods for pretraining CNNs and
  transformer-based architectures. Recently, the rise of text-to-image generative
  models, particularly those using denoising diffusion in a latent space, has
  introduced a new class of foundational models trained on massive, captioned
  image datasets. These models' ability to generate realistic images of unseen
  content suggests they possess a deep understanding of the visual world. In this
  work, we present Marigold, a family of conditional generative models and a
  fine-tuning protocol that extracts the knowledge from pretrained latent
  diffusion models like Stable Diffusion and adapts them for dense image analysis
  tasks, including monocular depth estimation, surface normals prediction, and
  intrinsic decomposition. Marigold requires minimal modification of the
  pre-trained latent diffusion model's architecture, trains with small synthetic
  datasets on a single GPU over a few days, and demonstrates state-of-the-art
  zero-shot generalization. Project page:
  https://marigoldcomputervision.github.io
paper_ai_summary: Marigold, a family of conditional generative models, leverages pretrained latent diffusion models for dense image analysis tasks with minimal modification and minimal data.
paper_ai_keywords: denoising diffusion latent diffusion models Stable Diffusion zero-shot generalization