De-Diffusion Makes Text a Strong Cross-Modal Interface
We demonstrate text as a strong cross-modal interface. Rather than relying on
deep embeddings to connect image and language as the interface representation,
our approach represents an image as text, from which we enjoy the
interpretability and flexibility inherent to natural language. We employ an
autoencoder that uses a pre-trained text-to-image diffusion model for decoding.
The encoder is trained to transform an input image into text, which is then fed
into the fixed text-to-image diffusion decoder to reconstruct the original
input -- a process we term De-Diffusion. Experiments validate both the
precision and comprehensiveness of De-Diffusion text representing images, such
that it can be readily ingested by off-the-shelf text-to-image tools and LLMs
for diverse multi-modal tasks. For example, a single De-Diffusion model can
generalize to provide transferable prompts for different text-to-image tools,
and also achieves a new state of the art on open-ended vision-language tasks by
simply prompting large language models with few-shot examples.
654304745670dd57b76ce7be
Chen Wei
claimed_verified
2023-11-03T09:02:06.614Z
6306bf8ddf993a789e6af8df
https://cdn-avatars.huggingface.co/v1/production/uploads/6306bf8ddf993a789e6af8df/d0VspqqEjQcDwCTFd0fM9.png
Chen Wei
user
weichen582
654304745670dd57b76ce7bf
Chenxi Liu
654304745670dd57b76ce7c0
Siyuan Qiao
654304745670dd57b76ce7c1
Zhishuai Zhang
654304745670dd57b76ce7c2
Alan Yuille
654304745670dd57b76ce7c3
Jiahui Yu
admin_assigned
2023-11-02T11:41:06.103Z
6174394801833e378dfc2517
/avatars/4989bcdda7ba1ae51cab8235c3e39275.svg
Jiahui Yu
user
jiahuiyu
De-Diffusion Makes Text a Strong Cross-Modal Interface
We demonstrate text as a strong cross-modal interface. Rather than relying on
deep embeddings to connect image and language as the interface representation,
our approach represents an image as text, from which we enjoy the
interpretability and flexibility inherent to natural language. We employ an
autoencoder that uses a pre-trained text-to-image diffusion model for decoding.
The encoder is trained to transform an input image into text, which is then fed
into the fixed text-to-image diffusion decoder to reconstruct the original
input -- a process we term De-Diffusion. Experiments validate both the
precision and comprehensiveness of De-Diffusion text representing images, such
that it can be readily ingested by off-the-shelf text-to-image tools and LLMs
for diverse multi-modal tasks. For example, a single De-Diffusion model can
generalize to provide transferable prompts for different text-to-image tools,
and also achieves a new state of the art on open-ended vision-language tasks by
simply prompting large language models with few-shot examples.
De-Diffusion uses a text-to-image diffusion model to represent images as text, enhancing interpretability and enabling transferable prompts for various vision-language tasks.
cross-modal interface deep embeddings text-to-image diffusion model autoencoder encoder De-Diffusion off-the-shelf text-to-image tools LLMs multi-modal tasks few-shot examples