title: Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision,
    Language, Audio, and Action
summary: We present Unified-IO 2, the first autoregressive multimodal model that is
  capable of understanding and generating image, text, audio, and action. To
  unify different modalities, we tokenize inputs and outputs -- images, text,
  audio, action, bounding boxes, etc., into a shared semantic space and then
  process them with a single encoder-decoder transformer model. Since training
  with such diverse modalities is challenging, we propose various architectural
  improvements to stabilize model training. We train our model from scratch on a
  large multimodal pre-training corpus from diverse sources with a multimodal
  mixture of denoisers objective. To learn an expansive set of skills, such as
  following multimodal instructions, we construct and finetune on an ensemble of
  120 datasets with prompts and augmentations. With a single unified model,
  Unified-IO 2 achieves state-of-the-art performance on the GRIT benchmark and
  strong results in more than 35 benchmarks, including image generation and
  understanding, natural language understanding, video and audio understanding,
  and robotic manipulation. We release all our models to the research community.
paper_authors._id: 658e40a016227c7a2dd07eb4
paper_authors.name: Jiasen Lu
paper_authors._id: 658e40a016227c7a2dd07eb5
paper_authors.name: Christopher Clark
paper_authors._id: 658e40a016227c7a2dd07eb6
paper_authors.name: Sangho Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:48:16.034Z
paper_authors._id: 664fc02d6f16bbd9a168654d
paper_authors.avatarUrl: /avatars/9f726ede9973706bc198845464238452.svg
paper_authors.fullname: Sangho Lee
paper_authors.type: user
paper_authors.user: sanghol
paper_authors._id: 658e40a016227c7a2dd07eb7
paper_authors.name: Zichen Zhang
paper_authors._id: 658e40a016227c7a2dd07eb8
paper_authors.name: Savya Khosla
paper_authors._id: 658e40a016227c7a2dd07eb9
paper_authors.name: Ryan Marten
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-16T20:17:32.793Z
paper_authors._id: 626c182a030a6e7363b6fe0a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626c182a030a6e7363b6fe0a/NaMcIOPVgJ6JFt8fojZWU.jpeg
paper_authors.fullname: Ryan Marten
paper_authors.type: user
paper_authors.user: ryanmarten
paper_authors._id: 658e40a016227c7a2dd07eba
paper_authors.name: Derek Hoiem
paper_authors._id: 658e40a016227c7a2dd07ebb
paper_authors.name: Aniruddha Kembhavi
paper_title: Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision,
    Language, Audio, and Action
paper_summary: We present Unified-IO 2, the first autoregressive multimodal model that is
  capable of understanding and generating image, text, audio, and action. To
  unify different modalities, we tokenize inputs and outputs -- images, text,
  audio, action, bounding boxes, etc., into a shared semantic space and then
  process them with a single encoder-decoder transformer model. Since training
  with such diverse modalities is challenging, we propose various architectural
  improvements to stabilize model training. We train our model from scratch on a
  large multimodal pre-training corpus from diverse sources with a multimodal
  mixture of denoisers objective. To learn an expansive set of skills, such as
  following multimodal instructions, we construct and finetune on an ensemble of
  120 datasets with prompts and augmentations. With a single unified model,
  Unified-IO 2 achieves state-of-the-art performance on the GRIT benchmark and
  strong results in more than 35 benchmarks, including image generation and
  understanding, natural language understanding, video and audio understanding,
  and robotic manipulation. We release all our models to the research community.
paper_ai_summary: A unified autoregressive multimodal model handles image, text, audio, and action by tokenizing inputs into a shared space and processing them with a transformer architecture, achieving state-of-the-art performance across various benchmarks.
paper_ai_keywords: autoregressive multimodal model encoder-decoder transformer multimodal mixture of denoisers prompts augmentations GRIT benchmark image generation natural language understanding video and audio understanding robotic manipulation