title: pOps: Photo-Inspired Diffusion Operators
summary: Text-guided image generation enables the creation of visual content from
  textual descriptions. However, certain visual concepts cannot be effectively
  conveyed through language alone. This has sparked a renewed interest in
  utilizing the CLIP image embedding space for more visually-oriented tasks
  through methods such as IP-Adapter. Interestingly, the CLIP image embedding
  space has been shown to be semantically meaningful, where linear operations
  within this space yield semantically meaningful results. Yet, the specific
  meaning of these operations can vary unpredictably across different images. To
  harness this potential, we introduce pOps, a framework that trains specific
  semantic operators directly on CLIP image embeddings. Each pOps operator is
  built upon a pretrained Diffusion Prior model. While the Diffusion Prior model
  was originally trained to map between text embeddings and image embeddings, we
  demonstrate that it can be tuned to accommodate new input conditions, resulting
  in a diffusion operator. Working directly over image embeddings not only
  improves our ability to learn semantic operations but also allows us to
  directly use a textual CLIP loss as an additional supervision when needed. We
  show that pOps can be used to learn a variety of photo-inspired operators with
  distinct semantic meanings, highlighting the semantic diversity and potential
  of our proposed approach.
paper_authors._id: 66625b29392d2f64ada3622f
paper_authors.name: Elad Richardson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-07T09:20:14.184Z
paper_authors._id: 633f370c13e836a0fc513724
paper_authors.avatarUrl: /avatars/38f3f9ecc5b9f35026becfb0c5c94fdf.svg
paper_authors.fullname: Elad Richardson
paper_authors.type: user
paper_authors.user: eladrich
paper_authors._id: 66625b29392d2f64ada36230
paper_authors.name: Yuval Alaluf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T12:49:38.572Z
paper_authors._id: 6310c1e0e87051f3e3e3bcc1
paper_authors.avatarUrl: /avatars/61b24a2eace7da0e27fbc4369a046051.svg
paper_authors.fullname: Yuval Alaluf
paper_authors.type: user
paper_authors.user: yuvalalaluf
paper_authors._id: 66625b29392d2f64ada36231
paper_authors.name: Ali Mahdavi-Amiri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T12:49:44.521Z
paper_authors._id: 64233145d654afd693193a17
paper_authors.avatarUrl: /avatars/9c668059e68773fab120f1a12c82f3ed.svg
paper_authors.fullname: Ali Mahdavi-Amiri
paper_authors.type: user
paper_authors.user: amahdavi
paper_authors._id: 66625b29392d2f64ada36232
paper_authors.name: Daniel Cohen-Or
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T12:49:51.731Z
paper_authors._id: 628507161949ebcae8e24ec3
paper_authors.avatarUrl: /avatars/008ecb3daa4c8187b5f339f1176b3c39.svg
paper_authors.fullname: Daniel Cohen-Or
paper_authors.type: user
paper_authors.user: cohenor
paper_title: pOps: Photo-Inspired Diffusion Operators
paper_summary: Text-guided image generation enables the creation of visual content from
  textual descriptions. However, certain visual concepts cannot be effectively
  conveyed through language alone. This has sparked a renewed interest in
  utilizing the CLIP image embedding space for more visually-oriented tasks
  through methods such as IP-Adapter. Interestingly, the CLIP image embedding
  space has been shown to be semantically meaningful, where linear operations
  within this space yield semantically meaningful results. Yet, the specific
  meaning of these operations can vary unpredictably across different images. To
  harness this potential, we introduce pOps, a framework that trains specific
  semantic operators directly on CLIP image embeddings. Each pOps operator is
  built upon a pretrained Diffusion Prior model. While the Diffusion Prior model
  was originally trained to map between text embeddings and image embeddings, we
  demonstrate that it can be tuned to accommodate new input conditions, resulting
  in a diffusion operator. Working directly over image embeddings not only
  improves our ability to learn semantic operations but also allows us to
  directly use a textual CLIP loss as an additional supervision when needed. We
  show that pOps can be used to learn a variety of photo-inspired operators with
  distinct semantic meanings, highlighting the semantic diversity and potential
  of our proposed approach.
paper_ai_keywords: CLIP IP-Adapter CLIP image embedding space semantic operators pOps Diffusion Prior model textual CLIP loss photo-inspired operators