title: CleanDIFT: Diffusion Features without Noise
summary: Internal features from large-scale pre-trained diffusion models have recently
  been established as powerful semantic descriptors for a wide range of
  downstream tasks. Works that use these features generally need to add noise to
  images before passing them through the model to obtain the semantic features,
  as the models do not offer the most useful features when given images with
  little to no noise. We show that this noise has a critical impact on the
  usefulness of these features that cannot be remedied by ensembling with
  different random noises. We address this issue by introducing a lightweight,
  unsupervised fine-tuning method that enables diffusion backbones to provide
  high-quality, noise-free semantic features. We show that these features readily
  outperform previous diffusion features by a wide margin in a wide variety of
  extraction setups and downstream tasks, offering better performance than even
  ensemble-based methods at a fraction of the cost.
paper_authors._id: 67516be84af3ce83f1e7dc67
paper_authors.name: Nick Stracke
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:44:40.411Z
paper_authors._id: 63a1c8d63c003e409311e3c3
paper_authors.avatarUrl: /avatars/c6cd546874abd5ee68ed88eff5ee1aa8.svg
paper_authors.fullname: Nick Stracke
paper_authors.type: user
paper_authors.user: kliyer
paper_authors._id: 67516be84af3ce83f1e7dc68
paper_authors.name: Stefan Andreas Baumann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:45:04.279Z
paper_authors._id: 6471c12a0c2b5fdaf1f07c45
paper_authors.avatarUrl: /avatars/6783067212d24d1e716a8b4c64df61b4.svg
paper_authors.fullname: Stefan Baumann
paper_authors.type: user
paper_authors.user: stefan-baumann
paper_authors._id: 67516be84af3ce83f1e7dc69
paper_authors.name: Kolja Bauer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:45:11.300Z
paper_authors._id: 65ae67bcbff3ae9f331c846a
paper_authors.avatarUrl: /avatars/6f9572fbfdba105638112386cdc346fa.svg
paper_authors.fullname: Kolja Bauer
paper_authors.type: user
paper_authors.user: kolja-b
paper_authors._id: 67516be84af3ce83f1e7dc6a
paper_authors.name: Frank Fundel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:45:17.457Z
paper_authors._id: 623dd9817d2f5b63ac1b73df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648220555715-623dd9817d2f5b63ac1b73df.jpeg
paper_authors.fullname: Frank Fundel
paper_authors.type: user
paper_authors.user: FrankFundel
paper_authors._id: 67516be84af3ce83f1e7dc6b
paper_authors.name: Björn Ommer
paper_title: CleanDIFT: Diffusion Features without Noise
paper_summary: Internal features from large-scale pre-trained diffusion models have recently
  been established as powerful semantic descriptors for a wide range of
  downstream tasks. Works that use these features generally need to add noise to
  images before passing them through the model to obtain the semantic features,
  as the models do not offer the most useful features when given images with
  little to no noise. We show that this noise has a critical impact on the
  usefulness of these features that cannot be remedied by ensembling with
  different random noises. We address this issue by introducing a lightweight,
  unsupervised fine-tuning method that enables diffusion backbones to provide
  high-quality, noise-free semantic features. We show that these features readily
  outperform previous diffusion features by a wide margin in a wide variety of
  extraction setups and downstream tasks, offering better performance than even
  ensemble-based methods at a fraction of the cost.
paper_ai_keywords: diffusion models semantic descriptors downstream tasks noise ensembling random noises lightweight unsupervised fine-tuning diffusion backbones noise-free semantic features