title: Generative Pretraining in Multimodality
summary: We present Emu, a Transformer-based multimodal foundation model, which can
  seamlessly generate images and texts in multimodal context. This omnivore model
  can take in any single-modality or multimodal data input indiscriminately
  (e.g., interleaved image, text and video) through a one-model-for-all
  autoregressive training process. First, visual signals are encoded into
  embeddings, and together with text tokens form an interleaved input sequence.
  Emu is then end-to-end trained with a unified objective of classifying the next
  text token or regressing the next visual embedding in the multimodal sequence.
  This versatile multimodality empowers the exploration of diverse pretraining
  data sources at scale, such as videos with interleaved frames and text,
  webpages with interleaved images and text, as well as web-scale image-text
  pairs and video-text pairs. Emu can serve as a generalist multimodal interface
  for both image-to-text and text-to-image tasks, and supports in-context image
  and text generation. Across a broad range of zero-shot/few-shot tasks including
  image captioning, visual question answering, video question answering and
  text-to-image generation, Emu demonstrates superb performance compared to
  state-of-the-art large multimodal models. Extended capabilities such as
  multimodal assistants via instruction tuning are also demonstrated with
  impressive performance.
paper_authors._id: 64adf672b7d86b40fd9400f1
paper_authors.name: Quan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T10:00:01.189Z
paper_authors._id: 630d7a8f81ef9b1772b67f4c
paper_authors.avatarUrl: /avatars/00757abd6e548ccebb5bfb233be129a2.svg
paper_authors.fullname: Quan Sun
paper_authors.type: user
paper_authors.user: QuanSun
paper_authors._id: 64adf672b7d86b40fd9400f2
paper_authors.name: Qiying Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T10:00:30.357Z
paper_authors._id: 6341375be5071b0c5cc946a3
paper_authors.avatarUrl: /avatars/c26a32b54a3f7731854059d247b5c523.svg
paper_authors.fullname: Qiying Yu
paper_authors.type: user
paper_authors.user: qiying
paper_authors._id: 64adf672b7d86b40fd9400f3
paper_authors.name: Yufeng Cui
paper_authors._id: 64adf672b7d86b40fd9400f4
paper_authors.name: Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:22:59.868Z
paper_authors._id: 640ed40dc025ddf618950af7
paper_authors.avatarUrl: /avatars/a99abd4346fa649ad4144f284ebcc972.svg
paper_authors.fullname: Fan Zhang
paper_authors.type: user
paper_authors.user: ryanzhangfan
paper_authors._id: 64adf672b7d86b40fd9400f5
paper_authors.name: Xiaosong Zhang
paper_authors._id: 64adf672b7d86b40fd9400f6
paper_authors.name: Yueze Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-27T07:00:08.681Z
paper_authors._id: 6458b59c7a7e192202df8fa0
paper_authors.avatarUrl: /avatars/33ee716477e5686da8723d01e199cd27.svg
paper_authors.fullname: Yueze Wang
paper_authors.type: user
paper_authors.user: yzwang
paper_authors._id: 64adf672b7d86b40fd9400f7
paper_authors.name: Hongcheng Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-12T13:28:59.021Z
paper_authors._id: 62728f4f6253fe2068da1021
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62728f4f6253fe2068da1021/KZ65X0EH98AF3zXemPiap.jpeg
paper_authors.fullname: Hongcheng Gao
paper_authors.type: user
paper_authors.user: HongchengGao
paper_authors._id: 64adf672b7d86b40fd9400f8
paper_authors.name: Jingjing Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-18T06:44:51.136Z
paper_authors._id: 64b5fb71fc5a8ae2a7e6af66
paper_authors.avatarUrl: /avatars/55fb935885f887d59880ddadb11f8cb8.svg
paper_authors.fullname: JJ Liu
paper_authors.type: user
paper_authors.user: sayitwithflower
paper_authors._id: 64adf672b7d86b40fd9400f9
paper_authors.name: Tiejun Huang
paper_authors._id: 64adf672b7d86b40fd9400fa
paper_authors.name: Xinlong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-12T13:28:55.348Z
paper_authors._id: 63ca558304c979828311c5a5
paper_authors.avatarUrl: /avatars/2a439d79fba2f987cabe780d10c94d25.svg
paper_authors.fullname: Xinlong Wang
paper_authors.type: user
paper_authors.user: xinlongwang
paper_title: Generative Pretraining in Multimodality
paper_summary: We present Emu, a Transformer-based multimodal foundation model, which can
  seamlessly generate images and texts in multimodal context. This omnivore model
  can take in any single-modality or multimodal data input indiscriminately
  (e.g., interleaved image, text and video) through a one-model-for-all
  autoregressive training process. First, visual signals are encoded into
  embeddings, and together with text tokens form an interleaved input sequence.
  Emu is then end-to-end trained with a unified objective of classifying the next
  text token or regressing the next visual embedding in the multimodal sequence.
  This versatile multimodality empowers the exploration of diverse pretraining
  data sources at scale, such as videos with interleaved frames and text,
  webpages with interleaved images and text, as well as web-scale image-text
  pairs and video-text pairs. Emu can serve as a generalist multimodal interface
  for both image-to-text and text-to-image tasks, and supports in-context image
  and text generation. Across a broad range of zero-shot/few-shot tasks including
  image captioning, visual question answering, video question answering and
  text-to-image generation, Emu demonstrates superb performance compared to
  state-of-the-art large multimodal models. Extended capabilities such as
  multimodal assistants via instruction tuning are also demonstrated with
  impressive performance.
paper_ai_keywords: Transformer-based multimodal model autoregressive training visual signals embeddings text tokens interleaved input sequence unified objective classifying the next text token regressing the next visual embedding multimodal sequence interleaved frames webpages with interleaved images and text web-scale image-text pairs video-text pairs image-to-text tasks text-to-image tasks zero-shot tasks few-shot tasks image captioning visual question answering video question answering text-to-image generation instruction tuning state-of-the-art large multimodal models multimodal assistants