Deconstructing Denoising Diffusion Models for Self-Supervised Learning
In this study, we examine the representation learning abilities of Denoising
Diffusion Models (DDM) that were originally purposed for image generation. Our
philosophy is to deconstruct a DDM, gradually transforming it into a classical
Denoising Autoencoder (DAE). This deconstructive procedure allows us to explore
how various components of modern DDMs influence self-supervised representation
learning. We observe that only a very few modern components are critical for
learning good representations, while many others are nonessential. Our study
ultimately arrives at an approach that is highly simplified and to a large
extent resembles a classical DAE. We hope our study will rekindle interest in a
family of classical methods within the realm of modern self-supervised
learning.
65b322613a32309ad7cf12f9
Xinlei Chen
admin_assigned
2024-01-26T09:32:20.277Z
63e58e3a006a775275e59e41
/avatars/75262a35b27a2ae1939df9118120d99e.svg
Xinlei Chen
user
endernewton
65b322613a32309ad7cf12fa
Zhuang Liu
claimed_verified
2024-02-21T19:37:59.921Z
647d3a9383cdb93baf76dce2
/avatars/56efca912a66ef9d4f2f3679aac31bae.svg
Zhuang Liu
user
liuzhuang13
65b322613a32309ad7cf12fb
Saining Xie
admin_assigned
2024-01-26T09:33:14.393Z
6596422646624a86ff3b3bda
/avatars/216e12b77e45ac5f1fa20932f5745411.svg
Saining Xie
user
sainx
65b322613a32309ad7cf12fc
Kaiming He
Deconstructing Denoising Diffusion Models for Self-Supervised Learning
In this study, we examine the representation learning abilities of Denoising
Diffusion Models (DDM) that were originally purposed for image generation. Our
philosophy is to deconstruct a DDM, gradually transforming it into a classical
Denoising Autoencoder (DAE). This deconstructive procedure allows us to explore
how various components of modern DDMs influence self-supervised representation
learning. We observe that only a very few modern components are critical for
learning good representations, while many others are nonessential. Our study
ultimately arrives at an approach that is highly simplified and to a large
extent resembles a classical DAE. We hope our study will rekindle interest in a
family of classical methods within the realm of modern self-supervised
learning.
A study on Denoising Diffusion Models reveals that only a few modern components are essential for self-supervised representation learning, leading to a simplified model resembling a Denoising Autoencoder.
Denoising Diffusion Models DDM Denoising Autoencoder DAE self-supervised representation learning