title: De-Diffusion Makes Text a Strong Cross-Modal Interface
summary: We demonstrate text as a strong cross-modal interface. Rather than relying on
  deep embeddings to connect image and language as the interface representation,
  our approach represents an image as text, from which we enjoy the
  interpretability and flexibility inherent to natural language. We employ an
  autoencoder that uses a pre-trained text-to-image diffusion model for decoding.
  The encoder is trained to transform an input image into text, which is then fed
  into the fixed text-to-image diffusion decoder to reconstruct the original
  input -- a process we term De-Diffusion. Experiments validate both the
  precision and comprehensiveness of De-Diffusion text representing images, such
  that it can be readily ingested by off-the-shelf text-to-image tools and LLMs
  for diverse multi-modal tasks. For example, a single De-Diffusion model can
  generalize to provide transferable prompts for different text-to-image tools,
  and also achieves a new state of the art on open-ended vision-language tasks by
  simply prompting large language models with few-shot examples.
paper_authors._id: 654304745670dd57b76ce7be
paper_authors.name: Chen Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-03T09:02:06.614Z
paper_authors._id: 6306bf8ddf993a789e6af8df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6306bf8ddf993a789e6af8df/d0VspqqEjQcDwCTFd0fM9.png
paper_authors.fullname: Chen Wei
paper_authors.type: user
paper_authors.user: weichen582
paper_authors._id: 654304745670dd57b76ce7bf
paper_authors.name: Chenxi Liu
paper_authors._id: 654304745670dd57b76ce7c0
paper_authors.name: Siyuan Qiao
paper_authors._id: 654304745670dd57b76ce7c1
paper_authors.name: Zhishuai Zhang
paper_authors._id: 654304745670dd57b76ce7c2
paper_authors.name: Alan Yuille
paper_authors._id: 654304745670dd57b76ce7c3
paper_authors.name: Jiahui Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T11:41:06.103Z
paper_authors._id: 6174394801833e378dfc2517
paper_authors.avatarUrl: /avatars/4989bcdda7ba1ae51cab8235c3e39275.svg
paper_authors.fullname: Jiahui Yu
paper_authors.type: user
paper_authors.user: jiahuiyu
paper_title: De-Diffusion Makes Text a Strong Cross-Modal Interface
paper_summary: We demonstrate text as a strong cross-modal interface. Rather than relying on
  deep embeddings to connect image and language as the interface representation,
  our approach represents an image as text, from which we enjoy the
  interpretability and flexibility inherent to natural language. We employ an
  autoencoder that uses a pre-trained text-to-image diffusion model for decoding.
  The encoder is trained to transform an input image into text, which is then fed
  into the fixed text-to-image diffusion decoder to reconstruct the original
  input -- a process we term De-Diffusion. Experiments validate both the
  precision and comprehensiveness of De-Diffusion text representing images, such
  that it can be readily ingested by off-the-shelf text-to-image tools and LLMs
  for diverse multi-modal tasks. For example, a single De-Diffusion model can
  generalize to provide transferable prompts for different text-to-image tools,
  and also achieves a new state of the art on open-ended vision-language tasks by
  simply prompting large language models with few-shot examples.
paper_ai_summary: De-Diffusion uses a text-to-image diffusion model to represent images as text, enhancing interpretability and enabling transferable prompts for various vision-language tasks.
paper_ai_keywords: cross-modal interface deep embeddings text-to-image diffusion model autoencoder encoder De-Diffusion off-the-shelf text-to-image tools LLMs multi-modal tasks few-shot examples