Generating Images with Multimodal Language Models
We propose a method to fuse frozen text-only large language models (LLMs)
with pre-trained image encoder and decoder models, by mapping between their
embedding spaces. Our model demonstrates a wide suite of multimodal
capabilities: image retrieval, novel image generation, and multimodal dialogue.
Ours is the first approach capable of conditioning on arbitrarily interleaved
image and text inputs to generate coherent image (and text) outputs. To achieve
strong performance on image generation, we propose an efficient mapping network
to ground the LLM to an off-the-shelf text-to-image generation model. This
mapping network translates hidden representations of text into the embedding
space of the visual models, enabling us to leverage the strong text
representations of the LLM for visual outputs. Our approach outperforms
baseline generation models on tasks with longer and more complex language. In
addition to novel image generation, our model is also capable of image
retrieval from a prespecified dataset, and decides whether to retrieve or
generate at inference time. This is done with a learnt decision module which
conditions on the hidden representations of the LLM. Our model exhibits a wider
range of capabilities compared to prior multimodal language models. It can
process image-and-text inputs, and produce retrieved images, generated images,
and generated text -- outperforming non-LLM based generation models across
several text-to-image tasks that measure context dependence.
6475777522b4dec4524c79b9
Jing Yu Koh
claimed_verified
2023-05-30T14:15:09.741Z
63877741189d6915f1d6291e
/avatars/3110962a08c4613f0f33a4f6d0ff69ce.svg
Jing Yu Koh
user
jykoh
6475777522b4dec4524c79ba
Daniel Fried
6475777522b4dec4524c79bb
Ruslan Salakhutdinov
Generating Images with Multimodal Language Models
We propose a method to fuse frozen text-only large language models (LLMs)
with pre-trained image encoder and decoder models, by mapping between their
embedding spaces. Our model demonstrates a wide suite of multimodal
capabilities: image retrieval, novel image generation, and multimodal dialogue.
Ours is the first approach capable of conditioning on arbitrarily interleaved
image and text inputs to generate coherent image (and text) outputs. To achieve
strong performance on image generation, we propose an efficient mapping network
to ground the LLM to an off-the-shelf text-to-image generation model. This
mapping network translates hidden representations of text into the embedding
space of the visual models, enabling us to leverage the strong text
representations of the LLM for visual outputs. Our approach outperforms
baseline generation models on tasks with longer and more complex language. In
addition to novel image generation, our model is also capable of image
retrieval from a prespecified dataset, and decides whether to retrieve or
generate at inference time. This is done with a learnt decision module which
conditions on the hidden representations of the LLM. Our model exhibits a wider
range of capabilities compared to prior multimodal language models. It can
process image-and-text inputs, and produce retrieved images, generated images,
and generated text -- outperforming non-LLM based generation models across
several text-to-image tasks that measure context dependence.
A method to integrate text-only large language models with image encoders and decoders through embedding space mapping achieves multimodal capabilities including image retrieval, generation, and dialogue.
large language models LLMs image encoder image decoder embedding spaces multimodal capabilities image retrieval novel image generation multimodal dialogue mapping network text-to-image generation model hidden representations visual models learned decision module multimodal language models