title: The Hidden Language of Diffusion Models
summary: Text-to-image diffusion models have demonstrated an unparalleled ability to
  generate high-quality, diverse images from a textual concept (e.g., "a doctor",
  "love"). However, the internal process of mapping text to a rich visual
  representation remains an enigma. In this work, we tackle the challenge of
  understanding concept representations in text-to-image models by decomposing an
  input text prompt into a small set of interpretable elements. This is achieved
  by learning a pseudo-token that is a sparse weighted combination of tokens from
  the model's vocabulary, with the objective of reconstructing the images
  generated for the given concept. Applied over the state-of-the-art Stable
  Diffusion model, this decomposition reveals non-trivial and surprising
  structures in the representations of concepts. For example, we find that some
  concepts such as "a president" or "a composer" are dominated by specific
  instances (e.g., "Obama", "Biden") and their interpolations. Other concepts,
  such as "happiness" combine associated terms that can be concrete ("family",
  "laughter") or abstract ("friendship", "emotion"). In addition to peering into
  the inner workings of Stable Diffusion, our method also enables applications
  such as single-image decomposition to tokens, bias detection and mitigation,
  and semantic image manipulation. Our code will be available at:
  https://hila-chefer.github.io/Conceptor/
paper_authors._id: 64796c09e504380e5c896730
paper_authors.name: Hila Chefer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-02T07:16:50.696Z
paper_authors._id: 6181c72cdcc1df2c9de8a4d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1655248010394-6181c72cdcc1df2c9de8a4d8.jpeg
paper_authors.fullname: Hila Chefer
paper_authors.type: user
paper_authors.user: Hila
paper_authors._id: 64796c09e504380e5c896731
paper_authors.name: Oran Lang
paper_authors._id: 64796c09e504380e5c896732
paper_authors.name: Mor Geva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-05T12:03:58.348Z
paper_authors._id: 610b729f9da682cd54ad9adf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1628140189042-noauth.jpeg
paper_authors.fullname: Mor Geva
paper_authors.type: user
paper_authors.user: mega
paper_authors._id: 64796c09e504380e5c896733
paper_authors.name: Volodymyr Polosukhin
paper_authors._id: 64796c09e504380e5c896734
paper_authors.name: Assaf Shocher
paper_authors._id: 64796c09e504380e5c896735
paper_authors.name: Michal Irani
paper_authors._id: 64796c09e504380e5c896736
paper_authors.name: Inbar Mosseri
paper_authors._id: 64796c09e504380e5c896737
paper_authors.name: Lior Wolf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-05T12:04:29.656Z
paper_authors._id: 62f9f170363251ee40a08046
paper_authors.avatarUrl: /avatars/67ad459db44b605be0c1f6bd33363d6e.svg
paper_authors.fullname: Lior Wolf
paper_authors.type: user
paper_authors.user: liorwolf
paper_title: The Hidden Language of Diffusion Models
paper_summary: Text-to-image diffusion models have demonstrated an unparalleled ability to
  generate high-quality, diverse images from a textual concept (e.g., "a doctor",
  "love"). However, the internal process of mapping text to a rich visual
  representation remains an enigma. In this work, we tackle the challenge of
  understanding concept representations in text-to-image models by decomposing an
  input text prompt into a small set of interpretable elements. This is achieved
  by learning a pseudo-token that is a sparse weighted combination of tokens from
  the model's vocabulary, with the objective of reconstructing the images
  generated for the given concept. Applied over the state-of-the-art Stable
  Diffusion model, this decomposition reveals non-trivial and surprising
  structures in the representations of concepts. For example, we find that some
  concepts such as "a president" or "a composer" are dominated by specific
  instances (e.g., "Obama", "Biden") and their interpolations. Other concepts,
  such as "happiness" combine associated terms that can be concrete ("family",
  "laughter") or abstract ("friendship", "emotion"). In addition to peering into
  the inner workings of Stable Diffusion, our method also enables applications
  such as single-image decomposition to tokens, bias detection and mitigation,
  and semantic image manipulation. Our code will be available at:
  https://hila-chefer.github.io/Conceptor/
paper_ai_keywords: diffusion models text-to-image pseudo-token vocabulary Stable Diffusion concept representations single-image decomposition bias detection semantic image manipulation