title: Emergent Correspondence from Image Diffusion
summary: Finding correspondences between images is a fundamental problem in computer
  vision. In this paper, we show that correspondence emerges in image diffusion
  models without any explicit supervision. We propose a simple strategy to
  extract this implicit knowledge out of diffusion networks as image features,
  namely DIffusion FeaTures (DIFT), and use them to establish correspondences
  between real images. Without any additional fine-tuning or supervision on the
  task-specific data or annotations, DIFT is able to outperform both
  weakly-supervised methods and competitive off-the-shelf features in identifying
  semantic, geometric, and temporal correspondences. Particularly for semantic
  correspondence, DIFT from Stable Diffusion is able to outperform DINO and
  OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k
  benchmark. It even outperforms the state-of-the-art supervised methods on 9 out
  of 18 categories while remaining on par for the overall performance. Project
  page: https://diffusionfeatures.github.io
paper_authors._id: 647fe3bcb82adfa7cc197ba0
paper_authors.name: Luming Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-09T15:34:52.635Z
paper_authors._id: 63e495a99db5da2dc1f679df
paper_authors.avatarUrl: /avatars/e3b3f7702a5c8fd21a246272940e45b8.svg
paper_authors.fullname: Luming Tang
paper_authors.type: user
paper_authors.user: lt453
paper_authors._id: 647fe3bcb82adfa7cc197ba1
paper_authors.name: Menglin Jia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-15T09:17:44.863Z
paper_authors._id: 648a4681fcda61143190645e
paper_authors.avatarUrl: /avatars/270bca51ef4d8453920713e9a33c46b3.svg
paper_authors.fullname: Menglin Jia
paper_authors.type: user
paper_authors.user: kmnp
paper_authors._id: 647fe3bcb82adfa7cc197ba2
paper_authors.name: Qianqian Wang
paper_authors._id: 647fe3bcb82adfa7cc197ba3
paper_authors.name: Cheng Perng Phoo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T14:04:41.144Z
paper_authors._id: 64020b3582bbdfe4b7c44edd
paper_authors.avatarUrl: /avatars/27c1d3eba6e89b4a56d1e171bffa8b80.svg
paper_authors.fullname: Cheng Perng Phoo
paper_authors.type: user
paper_authors.user: cpphoo
paper_authors._id: 647fe3bcb82adfa7cc197ba4
paper_authors.name: Bharath Hariharan
paper_title: Emergent Correspondence from Image Diffusion
paper_summary: Finding correspondences between images is a fundamental problem in computer
  vision. In this paper, we show that correspondence emerges in image diffusion
  models without any explicit supervision. We propose a simple strategy to
  extract this implicit knowledge out of diffusion networks as image features,
  namely DIffusion FeaTures (DIFT), and use them to establish correspondences
  between real images. Without any additional fine-tuning or supervision on the
  task-specific data or annotations, DIFT is able to outperform both
  weakly-supervised methods and competitive off-the-shelf features in identifying
  semantic, geometric, and temporal correspondences. Particularly for semantic
  correspondence, DIFT from Stable Diffusion is able to outperform DINO and
  OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k
  benchmark. It even outperforms the state-of-the-art supervised methods on 9 out
  of 18 categories while remaining on par for the overall performance. Project
  page: https://diffusionfeatures.github.io
paper_ai_keywords: diffusion models image diffusion models DIffusion FeaTures (DIFT) Stable Diffusion DINO OpenCLIP SPair-71k benchmark