pOps: Photo-Inspired Diffusion Operators
Text-guided image generation enables the creation of visual content from
textual descriptions. However, certain visual concepts cannot be effectively
conveyed through language alone. This has sparked a renewed interest in
utilizing the CLIP image embedding space for more visually-oriented tasks
through methods such as IP-Adapter. Interestingly, the CLIP image embedding
space has been shown to be semantically meaningful, where linear operations
within this space yield semantically meaningful results. Yet, the specific
meaning of these operations can vary unpredictably across different images. To
harness this potential, we introduce pOps, a framework that trains specific
semantic operators directly on CLIP image embeddings. Each pOps operator is
built upon a pretrained Diffusion Prior model. While the Diffusion Prior model
was originally trained to map between text embeddings and image embeddings, we
demonstrate that it can be tuned to accommodate new input conditions, resulting
in a diffusion operator. Working directly over image embeddings not only
improves our ability to learn semantic operations but also allows us to
directly use a textual CLIP loss as an additional supervision when needed. We
show that pOps can be used to learn a variety of photo-inspired operators with
distinct semantic meanings, highlighting the semantic diversity and potential
of our proposed approach.
66625b29392d2f64ada3622f
Elad Richardson
claimed_verified
2024-06-07T09:20:14.184Z
633f370c13e836a0fc513724
/avatars/38f3f9ecc5b9f35026becfb0c5c94fdf.svg
Elad Richardson
user
eladrich
66625b29392d2f64ada36230
Yuval Alaluf
admin_assigned
2024-06-07T12:49:38.572Z
6310c1e0e87051f3e3e3bcc1
/avatars/61b24a2eace7da0e27fbc4369a046051.svg
Yuval Alaluf
user
yuvalalaluf
66625b29392d2f64ada36231
Ali Mahdavi-Amiri
admin_assigned
2024-06-07T12:49:44.521Z
64233145d654afd693193a17
/avatars/9c668059e68773fab120f1a12c82f3ed.svg
Ali Mahdavi-Amiri
user
amahdavi
66625b29392d2f64ada36232
Daniel Cohen-Or
admin_assigned
2024-06-07T12:49:51.731Z
628507161949ebcae8e24ec3
/avatars/008ecb3daa4c8187b5f339f1176b3c39.svg
Daniel Cohen-Or
user
cohenor
pOps: Photo-Inspired Diffusion Operators
Text-guided image generation enables the creation of visual content from
textual descriptions. However, certain visual concepts cannot be effectively
conveyed through language alone. This has sparked a renewed interest in
utilizing the CLIP image embedding space for more visually-oriented tasks
through methods such as IP-Adapter. Interestingly, the CLIP image embedding
space has been shown to be semantically meaningful, where linear operations
within this space yield semantically meaningful results. Yet, the specific
meaning of these operations can vary unpredictably across different images. To
harness this potential, we introduce pOps, a framework that trains specific
semantic operators directly on CLIP image embeddings. Each pOps operator is
built upon a pretrained Diffusion Prior model. While the Diffusion Prior model
was originally trained to map between text embeddings and image embeddings, we
demonstrate that it can be tuned to accommodate new input conditions, resulting
in a diffusion operator. Working directly over image embeddings not only
improves our ability to learn semantic operations but also allows us to
directly use a textual CLIP loss as an additional supervision when needed. We
show that pOps can be used to learn a variety of photo-inspired operators with
distinct semantic meanings, highlighting the semantic diversity and potential
of our proposed approach.
pOps framework uses Diffusion Prior model to train semantic operators in CLIP image embedding space for text-guided image generation.
CLIP image embedding space IP-Adapter linear operations pOps Diffusion Prior model diffusion operator textual CLIP loss photo-inspired operators