Marigold: Affordable Adaptation of Diffusion-Based Image Generators for
  Image Analysis
The success of deep learning in computer vision over the past decade has
hinged on large labeled datasets and strong pretrained models. In data-scarce
settings, the quality of these pretrained models becomes crucial for effective
transfer learning. Image classification and self-supervised learning have
traditionally been the primary methods for pretraining CNNs and
transformer-based architectures. Recently, the rise of text-to-image generative
models, particularly those using denoising diffusion in a latent space, has
introduced a new class of foundational models trained on massive, captioned
image datasets. These models' ability to generate realistic images of unseen
content suggests they possess a deep understanding of the visual world. In this
work, we present Marigold, a family of conditional generative models and a
fine-tuning protocol that extracts the knowledge from pretrained latent
diffusion models like Stable Diffusion and adapts them for dense image analysis
tasks, including monocular depth estimation, surface normals prediction, and
intrinsic decomposition. Marigold requires minimal modification of the
pre-trained latent diffusion model's architecture, trains with small synthetic
datasets on a single GPU over a few days, and demonstrates state-of-the-art
zero-shot generalization. Project page:
https://marigoldcomputervision.github.io
6825c0e6bfd4908da7747c41
Bingxin Ke
admin_assigned
2025-05-15T13:55:42.142Z
63d90391da4f72339244c2a8
/avatars/eb0e0259c391d59739c1a205c36bb539.svg
Bingxin Ke
user
Bingxin
6825c0e6bfd4908da7747c42
Kevin Qu
6825c0e6bfd4908da7747c43
Tianfu Wang
claimed_verified
2025-05-16T07:12:38.377Z
633c25e000732349209d4e79
/avatars/0fd853f6cbfcacc98035364585d58f2c.svg
Tianfu Wang
user
Tianfwang
6825c0e6bfd4908da7747c44
Nando Metzger
claimed_verified
2025-05-15T13:45:09.533Z
63a5785a8fb23d08bb2d0291
/avatars/758b06dae06e9eee6fced10ce682aef1.svg
Nando Metzger
user
nandometzger
6825c0e6bfd4908da7747c45
Shengyu Huang
6825c0e6bfd4908da7747c46
Bo Li
claimed_verified
2025-05-21T09:03:44.866Z
6591e8f6ac02633c0dcb2020
/avatars/5a6e4db5bd3496948f94f68a849673f6.svg
Bo Li
user
bobboli
6825c0e6bfd4908da7747c47
Anton Obukhov
claimed_verified
2025-05-15T13:45:11.907Z
62f93abbc4817cfc0756b6f8
https://cdn-avatars.huggingface.co/v1/production/uploads/62f93abbc4817cfc0756b6f8/rGYLaq-rmoJJYotkC1VXk.jpeg
Anton Obukhov
user
toshas
6825c0e6bfd4908da7747c48
Konrad Schindler
extracted_confirmed
2025-05-15T10:53:52.126Z
6750649b7edd6a98a1bbcd06
/avatars/ba27f12d0333cf2d400d4405af7efe97.svg
Konrad Schindler
user
konradschindler
Marigold: Affordable Adaptation of Diffusion-Based Image Generators for
  Image Analysis
The success of deep learning in computer vision over the past decade has
hinged on large labeled datasets and strong pretrained models. In data-scarce
settings, the quality of these pretrained models becomes crucial for effective
transfer learning. Image classification and self-supervised learning have
traditionally been the primary methods for pretraining CNNs and
transformer-based architectures. Recently, the rise of text-to-image generative
models, particularly those using denoising diffusion in a latent space, has
introduced a new class of foundational models trained on massive, captioned
image datasets. These models' ability to generate realistic images of unseen
content suggests they possess a deep understanding of the visual world. In this
work, we present Marigold, a family of conditional generative models and a
fine-tuning protocol that extracts the knowledge from pretrained latent
diffusion models like Stable Diffusion and adapts them for dense image analysis
tasks, including monocular depth estimation, surface normals prediction, and
intrinsic decomposition. Marigold requires minimal modification of the
pre-trained latent diffusion model's architecture, trains with small synthetic
datasets on a single GPU over a few days, and demonstrates state-of-the-art
zero-shot generalization. Project page:
https://marigoldcomputervision.github.io
Marigold, a family of conditional generative models, leverages pretrained latent diffusion models for dense image analysis tasks with minimal modification and minimal data.
denoising diffusion latent diffusion models Stable Diffusion zero-shot generalization