title: MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data
summary: We train a model to generate images from multimodal prompts of interleaved
  text and images such as "a <picture of a man> man and his <picture of a dog>
  dog in an <picture of a cartoon> animated style." We bootstrap a multimodal
  dataset by extracting semantically meaningful image crops corresponding to
  words in the image captions of synthetically generated and publicly available
  text-image data. Our model, MUMU, is composed of a vision-language model
  encoder with a diffusion decoder and is trained on a single 8xH100 GPU node.
  Despite being only trained on crops from the same image, MUMU learns to compose
  inputs from different images into a coherent output. For example, an input of a
  realistic person and a cartoon will output the same person in the cartoon
  style, and an input of a standing subject and a scooter will output the subject
  riding the scooter. As a result, our model generalizes to tasks such as style
  transfer and character consistency. Our results show the promise of using
  multimodal models as general purpose controllers for image generation.
paper_authors._id: 667ec70aa7d8b1deba80afa8
paper_authors.name: William Berman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T14:57:47.457Z
paper_authors._id: 63407fadb78ed99eab00203d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668468846504-63407fadb78ed99eab00203d.jpeg
paper_authors.fullname: Will Berman
paper_authors.type: user
paper_authors.user: williamberman
paper_authors._id: 667ec70aa7d8b1deba80afa9
paper_authors.name: Alexander Peysakhovich
paper_title: MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data
paper_summary: We train a model to generate images from multimodal prompts of interleaved
  text and images such as "a <picture of a man> man and his <picture of a dog>
  dog in an <picture of a cartoon> animated style." We bootstrap a multimodal
  dataset by extracting semantically meaningful image crops corresponding to
  words in the image captions of synthetically generated and publicly available
  text-image data. Our model, MUMU, is composed of a vision-language model
  encoder with a diffusion decoder and is trained on a single 8xH100 GPU node.
  Despite being only trained on crops from the same image, MUMU learns to compose
  inputs from different images into a coherent output. For example, an input of a
  realistic person and a cartoon will output the same person in the cartoon
  style, and an input of a standing subject and a scooter will output the subject
  riding the scooter. As a result, our model generalizes to tasks such as style
  transfer and character consistency. Our results show the promise of using
  multimodal models as general purpose controllers for image generation.
paper_ai_keywords: diffusion decoder vision-language model encoder multimodal dataset image crops image captions multimodal models