title: Jointly Training Large Autoregressive Multimodal Models
summary: In recent years, advances in the large-scale pretraining of language and
  text-to-image models have revolutionized the field of machine learning. Yet,
  integrating these two modalities into a single, robust model capable of
  generating seamless multimodal outputs remains a significant challenge. To
  address this gap, we present the Joint Autoregressive Mixture (JAM) framework,
  a modular approach that systematically fuses existing text and image generation
  models. We also introduce a specialized, data-efficient instruction-tuning
  strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned
  model demonstrates unparalleled performance in generating high-quality
  multimodal outputs and represents the first model explicitly designed for this
  purpose.
paper_authors._id: 651501516a451f1caf345611
paper_authors.name: Emanuele Aiello
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-15T21:28:49.848Z
paper_authors._id: 6515ac7f60757b8c8f6b8d31
paper_authors.avatarUrl: /avatars/ed033f30f9f78ed68da976e6d01af0ea.svg
paper_authors.fullname: Emanuele Aiello
paper_authors.type: user
paper_authors.user: Ema97x
paper_authors._id: 651501516a451f1caf345612
paper_authors.name: Lili Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-28T09:19:30.861Z
paper_authors._id: 6466d867f3e78d1d6be05da0
paper_authors.avatarUrl: /avatars/4a2458f9fe5e1637798adfaf01cfa4fa.svg
paper_authors.fullname: Lili Yu
paper_authors.type: user
paper_authors.user: lilisierrayu
paper_authors._id: 651501516a451f1caf345613
paper_authors.name: Yixin Nie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-28T09:19:42.319Z
paper_authors._id: 5f7919d5f1e7ef6e919a1f55
paper_authors.avatarUrl: /avatars/fb747fb43b65ccdfb08d7bf71525f92f.svg
paper_authors.fullname: Yixin Nie
paper_authors.type: user
paper_authors.user: ynie
paper_authors._id: 651501516a451f1caf345614
paper_authors.name: Armen Aghajanyan
paper_authors.statusLastChangedAt: 2023-09-28T09:19:53.408Z
paper_authors._id: 651501516a451f1caf345615
paper_authors.name: Barlas Oguz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-28T09:19:59.971Z
paper_authors._id: 6317cb5b212fce5a3cd97227
paper_authors.avatarUrl: /avatars/c2b4eefa7c17cdf2299c07466c361ab0.svg
paper_authors.fullname: Barlas Oguz
paper_authors.type: user
paper_authors.user: barlaso
paper_title: Jointly Training Large Autoregressive Multimodal Models
paper_summary: In recent years, advances in the large-scale pretraining of language and
  text-to-image models have revolutionized the field of machine learning. Yet,
  integrating these two modalities into a single, robust model capable of
  generating seamless multimodal outputs remains a significant challenge. To
  address this gap, we present the Joint Autoregressive Mixture (JAM) framework,
  a modular approach that systematically fuses existing text and image generation
  models. We also introduce a specialized, data-efficient instruction-tuning
  strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned
  model demonstrates unparalleled performance in generating high-quality
  multimodal outputs and represents the first model explicitly designed for this
  purpose.
paper_ai_summary: The Joint Autoregressive Mixture (JAM) framework integrates text and image generation models for high-quality multimodal output, exceeding previous methods with a specialized instruction-tuning strategy.
paper_ai_keywords: Joint Autoregressive Mixture JAM framework multimodal outputs instruction-tuning text generation image generation