title: Generating Images with Multimodal Language Models
summary: We propose a method to fuse frozen text-only large language models (LLMs)
  with pre-trained image encoder and decoder models, by mapping between their
  embedding spaces. Our model demonstrates a wide suite of multimodal
  capabilities: image retrieval, novel image generation, and multimodal dialogue.
  Ours is the first approach capable of conditioning on arbitrarily interleaved
  image and text inputs to generate coherent image (and text) outputs. To achieve
  strong performance on image generation, we propose an efficient mapping network
  to ground the LLM to an off-the-shelf text-to-image generation model. This
  mapping network translates hidden representations of text into the embedding
  space of the visual models, enabling us to leverage the strong text
  representations of the LLM for visual outputs. Our approach outperforms
  baseline generation models on tasks with longer and more complex language. In
  addition to novel image generation, our model is also capable of image
  retrieval from a prespecified dataset, and decides whether to retrieve or
  generate at inference time. This is done with a learnt decision module which
  conditions on the hidden representations of the LLM. Our model exhibits a wider
  range of capabilities compared to prior multimodal language models. It can
  process image-and-text inputs, and produce retrieved images, generated images,
  and generated text -- outperforming non-LLM based generation models across
  several text-to-image tasks that measure context dependence.
paper_authors._id: 6475777522b4dec4524c79b9
paper_authors.name: Jing Yu Koh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T14:15:09.741Z
paper_authors._id: 63877741189d6915f1d6291e
paper_authors.avatarUrl: /avatars/3110962a08c4613f0f33a4f6d0ff69ce.svg
paper_authors.fullname: Jing Yu Koh
paper_authors.type: user
paper_authors.user: jykoh
paper_authors._id: 6475777522b4dec4524c79ba
paper_authors.name: Daniel Fried
paper_authors._id: 6475777522b4dec4524c79bb
paper_authors.name: Ruslan Salakhutdinov
paper_title: Generating Images with Multimodal Language Models
paper_summary: We propose a method to fuse frozen text-only large language models (LLMs)
  with pre-trained image encoder and decoder models, by mapping between their
  embedding spaces. Our model demonstrates a wide suite of multimodal
  capabilities: image retrieval, novel image generation, and multimodal dialogue.
  Ours is the first approach capable of conditioning on arbitrarily interleaved
  image and text inputs to generate coherent image (and text) outputs. To achieve
  strong performance on image generation, we propose an efficient mapping network
  to ground the LLM to an off-the-shelf text-to-image generation model. This
  mapping network translates hidden representations of text into the embedding
  space of the visual models, enabling us to leverage the strong text
  representations of the LLM for visual outputs. Our approach outperforms
  baseline generation models on tasks with longer and more complex language. In
  addition to novel image generation, our model is also capable of image
  retrieval from a prespecified dataset, and decides whether to retrieve or
  generate at inference time. This is done with a learnt decision module which
  conditions on the hidden representations of the LLM. Our model exhibits a wider
  range of capabilities compared to prior multimodal language models. It can
  process image-and-text inputs, and produce retrieved images, generated images,
  and generated text -- outperforming non-LLM based generation models across
  several text-to-image tasks that measure context dependence.
paper_ai_keywords: large language models (LLMs) pre-trained image encoder pre-trained image decoder models embedding spaces multimodal capabilities image retrieval image generation multimodal dialogue arbitrarily interleaved mapping network hidden representations visual models text-to-image generation model performance decision module context dependence