Unsupervised Semantic Correspondence Using Stable Diffusion
Text-to-image diffusion models are now capable of generating images that are
often indistinguishable from real images. To generate such images, these models
must understand the semantics of the objects they are asked to generate. In
this work we show that, without any training, one can leverage this semantic
knowledge within diffusion models to find semantic correspondences -- locations
in multiple images that have the same semantic meaning. Specifically, given an
image, we optimize the prompt embeddings of these models for maximum attention
on the regions of interest. These optimized embeddings capture semantic
information about the location, which can then be transferred to another image.
By doing so we obtain results on par with the strongly supervised state of the
art on the PF-Willow dataset and significantly outperform (20.9% relative for
the SPair-71k dataset) any existing weakly or unsupervised method on PF-Willow,
CUB-200 and SPair-71k datasets.
64700f767a64358741c609ca
Eric Hedlin
admin_assigned
2023-05-26T14:00:26.322Z
5ec71015968f6028e0559f6b
/avatars/6162834f788aaae52f49365c0ac778a0.svg
Eric Hedlin
user
iamerichedlin
64700f767a64358741c609cb
Gopal Sharma
64700f767a64358741c609cc
Shweta Mahajan
claimed_verified
2023-06-28T20:14:46.167Z
6428a10d88215cee63b6e0f8
/avatars/36159c93e5f956e82f4dbf69785be53e.svg
Shweta Mahajan
user
s-mahajan
64700f767a64358741c609cd
Hossam Isack
64700f767a64358741c609ce
Abhishek Kar
64700f767a64358741c609cf
Andrea Tagliasacchi
claimed_verified
2023-05-29T08:43:14.976Z
6310c97fa23f0327bce7f93d
/avatars/999c12348ce57d71f15337f50a6b2685.svg
Andrea
user
taiya
64700f767a64358741c609d0
Kwang Moo Yi
claimed_verified
2023-07-05T18:56:54.889Z
64a591af9b8ae9ccccfdd3e7
/avatars/c6cd43d814956261639fe594bc4aa181.svg
Kwang Moo Yi
user
kmyi
Unsupervised Semantic Correspondence Using Stable Diffusion
Text-to-image diffusion models are now capable of generating images that are
often indistinguishable from real images. To generate such images, these models
must understand the semantics of the objects they are asked to generate. In
this work we show that, without any training, one can leverage this semantic
knowledge within diffusion models to find semantic correspondences -- locations
in multiple images that have the same semantic meaning. Specifically, given an
image, we optimize the prompt embeddings of these models for maximum attention
on the regions of interest. These optimized embeddings capture semantic
information about the location, which can then be transferred to another image.
By doing so we obtain results on par with the strongly supervised state of the
art on the PF-Willow dataset and significantly outperform (20.9% relative for
the SPair-71k dataset) any existing weakly or unsupervised method on PF-Willow,
CUB-200 and SPair-71k datasets.
Text-to-image diffusion models leverage semantic knowledge to find correspondences between images without training, achieving state-of-the-art results on multiple datasets.
text-to-image diffusion models semantic knowledge prompt embeddings attention semantic correspondences PF-Willow dataset SPair-71k dataset CUB-200 dataset