title: DreamTeacher: Pretraining Image Backbones with Deep Generative Models
summary: In this work, we introduce a self-supervised feature representation learning
  framework DreamTeacher that utilizes generative networks for pre-training
  downstream image backbones. We propose to distill knowledge from a trained
  generative model into standard image backbones that have been well engineered
  for specific perception tasks. We investigate two types of knowledge
  distillation: 1) distilling learned generative features onto target image
  backbones as an alternative to pretraining these backbones on large labeled
  datasets such as ImageNet, and 2) distilling labels obtained from generative
  networks with task heads onto logits of target backbones. We perform extensive
  analyses on multiple generative models, dense prediction benchmarks, and
  several pre-training regimes. We empirically find that our DreamTeacher
  significantly outperforms existing self-supervised representation learning
  approaches across the board. Unsupervised ImageNet pre-training with
  DreamTeacher leads to significant improvements over ImageNet classification
  pre-training on downstream datasets, showcasing generative models, and
  diffusion generative models specifically, as a promising approach to
  representation learning on large, diverse datasets without requiring manual
  annotation.
paper_authors._id: 64b4aceeffed52962ad7a971
paper_authors.name: Daiqing Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T11:09:16.227Z
paper_authors._id: 633f44efc11d723b1809958b
paper_authors.avatarUrl: /avatars/d70b067deabddae984c3637290b7de99.svg
paper_authors.fullname: Li
paper_authors.type: user
paper_authors.user: Daiqing
paper_authors._id: 64b4aceeffed52962ad7a972
paper_authors.name: Huan Ling
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-18T06:44:12.036Z
paper_authors._id: 64a1910bb14ab77f9e46497a
paper_authors.avatarUrl: /avatars/5e6b63233274e89bf74d132eae03a374.svg
paper_authors.fullname: Huan
paper_authors.type: user
paper_authors.user: linghuan4545
paper_authors._id: 64b4aceeffed52962ad7a973
paper_authors.name: Amlan Kar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T11:09:44.598Z
paper_authors._id: 619ebcc653061ce00477b08d
paper_authors.avatarUrl: /avatars/f754e144accaacb8f18331d5d49a6a37.svg
paper_authors.fullname: Amlan Kar
paper_authors.type: user
paper_authors.user: amlankar
paper_authors._id: 64b4aceeffed52962ad7a974
paper_authors.name: David Acuna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T11:10:00.229Z
paper_authors._id: 645abb43c4acfcf6640270e3
paper_authors.avatarUrl: /avatars/e65f8de9b9fe878bd8e9106a802c6f21.svg
paper_authors.fullname: David Acuna
paper_authors.type: user
paper_authors.user: davidjesuscau
paper_authors._id: 64b4aceeffed52962ad7a975
paper_authors.name: Seung Wook Kim
paper_authors._id: 64b4aceeffed52962ad7a976
paper_authors.name: Karsten Kreis
paper_authors._id: 64b4aceeffed52962ad7a977
paper_authors.name: Antonio Torralba
paper_authors._id: 64b4aceeffed52962ad7a978
paper_authors.name: Sanja Fidler
paper_title: DreamTeacher: Pretraining Image Backbones with Deep Generative Models
paper_summary: In this work, we introduce a self-supervised feature representation learning
  framework DreamTeacher that utilizes generative networks for pre-training
  downstream image backbones. We propose to distill knowledge from a trained
  generative model into standard image backbones that have been well engineered
  for specific perception tasks. We investigate two types of knowledge
  distillation: 1) distilling learned generative features onto target image
  backbones as an alternative to pretraining these backbones on large labeled
  datasets such as ImageNet, and 2) distilling labels obtained from generative
  networks with task heads onto logits of target backbones. We perform extensive
  analyses on multiple generative models, dense prediction benchmarks, and
  several pre-training regimes. We empirically find that our DreamTeacher
  significantly outperforms existing self-supervised representation learning
  approaches across the board. Unsupervised ImageNet pre-training with
  DreamTeacher leads to significant improvements over ImageNet classification
  pre-training on downstream datasets, showcasing generative models, and
  diffusion generative models specifically, as a promising approach to
  representation learning on large, diverse datasets without requiring manual
  annotation.
paper_ai_summary: DreamTeacher uses generative networks to distill knowledge into self-supervised image backbones, improving performance across various benchmarks and datasets without labeled data.
paper_ai_keywords: self-supervised feature representation DreamTeacher generative networks pre-training dense prediction knowledge distillation ImageNet diffusion generative models representation learning