title: Deconstructing Denoising Diffusion Models for Self-Supervised Learning
summary: In this study, we examine the representation learning abilities of Denoising
  Diffusion Models (DDM) that were originally purposed for image generation. Our
  philosophy is to deconstruct a DDM, gradually transforming it into a classical
  Denoising Autoencoder (DAE). This deconstructive procedure allows us to explore
  how various components of modern DDMs influence self-supervised representation
  learning. We observe that only a very few modern components are critical for
  learning good representations, while many others are nonessential. Our study
  ultimately arrives at an approach that is highly simplified and to a large
  extent resembles a classical DAE. We hope our study will rekindle interest in a
  family of classical methods within the realm of modern self-supervised
  learning.
paper_authors._id: 65b322613a32309ad7cf12f9
paper_authors.name: Xinlei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T09:32:20.277Z
paper_authors._id: 63e58e3a006a775275e59e41
paper_authors.avatarUrl: /avatars/75262a35b27a2ae1939df9118120d99e.svg
paper_authors.fullname: Xinlei Chen
paper_authors.type: user
paper_authors.user: endernewton
paper_authors._id: 65b322613a32309ad7cf12fa
paper_authors.name: Zhuang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T19:37:59.921Z
paper_authors._id: 647d3a9383cdb93baf76dce2
paper_authors.avatarUrl: /avatars/56efca912a66ef9d4f2f3679aac31bae.svg
paper_authors.fullname: Zhuang Liu
paper_authors.type: user
paper_authors.user: liuzhuang13
paper_authors._id: 65b322613a32309ad7cf12fb
paper_authors.name: Saining Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T09:33:14.393Z
paper_authors._id: 6596422646624a86ff3b3bda
paper_authors.avatarUrl: /avatars/216e12b77e45ac5f1fa20932f5745411.svg
paper_authors.fullname: Saining Xie
paper_authors.type: user
paper_authors.user: sainx
paper_authors._id: 65b322613a32309ad7cf12fc
paper_authors.name: Kaiming He
paper_title: Deconstructing Denoising Diffusion Models for Self-Supervised Learning
paper_summary: In this study, we examine the representation learning abilities of Denoising
  Diffusion Models (DDM) that were originally purposed for image generation. Our
  philosophy is to deconstruct a DDM, gradually transforming it into a classical
  Denoising Autoencoder (DAE). This deconstructive procedure allows us to explore
  how various components of modern DDMs influence self-supervised representation
  learning. We observe that only a very few modern components are critical for
  learning good representations, while many others are nonessential. Our study
  ultimately arrives at an approach that is highly simplified and to a large
  extent resembles a classical DAE. We hope our study will rekindle interest in a
  family of classical methods within the realm of modern self-supervised
  learning.
paper_ai_summary: A study on Denoising Diffusion Models reveals that only a few modern components are essential for self-supervised representation learning, leading to a simplified model resembling a Denoising Autoencoder.
paper_ai_keywords: Denoising Diffusion Models DDM Denoising Autoencoder DAE self-supervised representation learning