title: Unsupervised Semantic Correspondence Using Stable Diffusion
summary: Text-to-image diffusion models are now capable of generating images that are
  often indistinguishable from real images. To generate such images, these models
  must understand the semantics of the objects they are asked to generate. In
  this work we show that, without any training, one can leverage this semantic
  knowledge within diffusion models to find semantic correspondences -- locations
  in multiple images that have the same semantic meaning. Specifically, given an
  image, we optimize the prompt embeddings of these models for maximum attention
  on the regions of interest. These optimized embeddings capture semantic
  information about the location, which can then be transferred to another image.
  By doing so we obtain results on par with the strongly supervised state of the
  art on the PF-Willow dataset and significantly outperform (20.9% relative for
  the SPair-71k dataset) any existing weakly or unsupervised method on PF-Willow,
  CUB-200 and SPair-71k datasets.
paper_authors._id: 64700f767a64358741c609ca
paper_authors.name: Eric Hedlin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T14:00:26.322Z
paper_authors._id: 5ec71015968f6028e0559f6b
paper_authors.avatarUrl: /avatars/6162834f788aaae52f49365c0ac778a0.svg
paper_authors.fullname: Eric Hedlin
paper_authors.type: user
paper_authors.user: iamerichedlin
paper_authors._id: 64700f767a64358741c609cb
paper_authors.name: Gopal Sharma
paper_authors._id: 64700f767a64358741c609cc
paper_authors.name: Shweta Mahajan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:14:46.167Z
paper_authors._id: 6428a10d88215cee63b6e0f8
paper_authors.avatarUrl: /avatars/36159c93e5f956e82f4dbf69785be53e.svg
paper_authors.fullname: Shweta Mahajan
paper_authors.type: user
paper_authors.user: s-mahajan
paper_authors._id: 64700f767a64358741c609cd
paper_authors.name: Hossam Isack
paper_authors._id: 64700f767a64358741c609ce
paper_authors.name: Abhishek Kar
paper_authors._id: 64700f767a64358741c609cf
paper_authors.name: Andrea Tagliasacchi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-29T08:43:14.976Z
paper_authors._id: 6310c97fa23f0327bce7f93d
paper_authors.avatarUrl: /avatars/999c12348ce57d71f15337f50a6b2685.svg
paper_authors.fullname: Andrea
paper_authors.type: user
paper_authors.user: taiya
paper_authors._id: 64700f767a64358741c609d0
paper_authors.name: Kwang Moo Yi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-05T18:56:54.889Z
paper_authors._id: 64a591af9b8ae9ccccfdd3e7
paper_authors.avatarUrl: /avatars/c6cd43d814956261639fe594bc4aa181.svg
paper_authors.fullname: Kwang Moo Yi
paper_authors.type: user
paper_authors.user: kmyi
paper_title: Unsupervised Semantic Correspondence Using Stable Diffusion
paper_summary: Text-to-image diffusion models are now capable of generating images that are
  often indistinguishable from real images. To generate such images, these models
  must understand the semantics of the objects they are asked to generate. In
  this work we show that, without any training, one can leverage this semantic
  knowledge within diffusion models to find semantic correspondences -- locations
  in multiple images that have the same semantic meaning. Specifically, given an
  image, we optimize the prompt embeddings of these models for maximum attention
  on the regions of interest. These optimized embeddings capture semantic
  information about the location, which can then be transferred to another image.
  By doing so we obtain results on par with the strongly supervised state of the
  art on the PF-Willow dataset and significantly outperform (20.9% relative for
  the SPair-71k dataset) any existing weakly or unsupervised method on PF-Willow,
  CUB-200 and SPair-71k datasets.
paper_ai_keywords: diffusion models prompt embeddings attention regions of interest semantic information locations prompt embeddings Discriminative correspond attentiveness transfer