Generative Pretraining in Multimodality
We present Emu, a Transformer-based multimodal foundation model, which can
seamlessly generate images and texts in multimodal context. This omnivore model
can take in any single-modality or multimodal data input indiscriminately
(e.g., interleaved image, text and video) through a one-model-for-all
autoregressive training process. First, visual signals are encoded into
embeddings, and together with text tokens form an interleaved input sequence.
Emu is then end-to-end trained with a unified objective of classifying the next
text token or regressing the next visual embedding in the multimodal sequence.
This versatile multimodality empowers the exploration of diverse pretraining
data sources at scale, such as videos with interleaved frames and text,
webpages with interleaved images and text, as well as web-scale image-text
pairs and video-text pairs. Emu can serve as a generalist multimodal interface
for both image-to-text and text-to-image tasks, and supports in-context image
and text generation. Across a broad range of zero-shot/few-shot tasks including
image captioning, visual question answering, video question answering and
text-to-image generation, Emu demonstrates superb performance compared to
state-of-the-art large multimodal models. Extended capabilities such as
multimodal assistants via instruction tuning are also demonstrated with
impressive performance.
64adf672b7d86b40fd9400f1
Quan Sun
admin_assigned
2023-07-13T10:00:01.189Z
630d7a8f81ef9b1772b67f4c
/avatars/00757abd6e548ccebb5bfb233be129a2.svg
Quan Sun
user
QuanSun
64adf672b7d86b40fd9400f2
Qiying Yu
admin_assigned
2023-07-13T10:00:30.357Z
6341375be5071b0c5cc946a3
/avatars/c26a32b54a3f7731854059d247b5c523.svg
Qiying Yu
user
qiying
64adf672b7d86b40fd9400f3
Yufeng Cui
64adf672b7d86b40fd9400f4
Fan Zhang
claimed_verified
2024-10-08T07:22:59.868Z
640ed40dc025ddf618950af7
/avatars/a99abd4346fa649ad4144f284ebcc972.svg
Fan Zhang
user
ryanzhangfan
64adf672b7d86b40fd9400f5
Xiaosong Zhang
64adf672b7d86b40fd9400f6
Yueze Wang
claimed_verified
2024-09-27T07:00:08.681Z
6458b59c7a7e192202df8fa0
/avatars/33ee716477e5686da8723d01e199cd27.svg
Yueze Wang
user
yzwang
64adf672b7d86b40fd9400f7
Hongcheng Gao
claimed_verified
2023-07-12T13:28:59.021Z
62728f4f6253fe2068da1021
https://cdn-avatars.huggingface.co/v1/production/uploads/62728f4f6253fe2068da1021/KZ65X0EH98AF3zXemPiap.jpeg
Hongcheng Gao
user
HongchengGao
64adf672b7d86b40fd9400f8
Jingjing Liu
claimed_verified
2023-07-18T06:44:51.136Z
64b5fb71fc5a8ae2a7e6af66
/avatars/55fb935885f887d59880ddadb11f8cb8.svg
JJ Liu
user
sayitwithflower
64adf672b7d86b40fd9400f9
Tiejun Huang
64adf672b7d86b40fd9400fa
Xinlong Wang
claimed_verified
2023-07-12T13:28:55.348Z
63ca558304c979828311c5a5
/avatars/2a439d79fba2f987cabe780d10c94d25.svg
Xinlong Wang
user
xinlongwang
Generative Pretraining in Multimodality
We present Emu, a Transformer-based multimodal foundation model, which can
seamlessly generate images and texts in multimodal context. This omnivore model
can take in any single-modality or multimodal data input indiscriminately
(e.g., interleaved image, text and video) through a one-model-for-all
autoregressive training process. First, visual signals are encoded into
embeddings, and together with text tokens form an interleaved input sequence.
Emu is then end-to-end trained with a unified objective of classifying the next
text token or regressing the next visual embedding in the multimodal sequence.
This versatile multimodality empowers the exploration of diverse pretraining
data sources at scale, such as videos with interleaved frames and text,
webpages with interleaved images and text, as well as web-scale image-text
pairs and video-text pairs. Emu can serve as a generalist multimodal interface
for both image-to-text and text-to-image tasks, and supports in-context image
and text generation. Across a broad range of zero-shot/few-shot tasks including
image captioning, visual question answering, video question answering and
text-to-image generation, Emu demonstrates superb performance compared to
state-of-the-art large multimodal models. Extended capabilities such as
multimodal assistants via instruction tuning are also demonstrated with
impressive performance.
Emu, a Transformer-based multimodal model, generates images and texts in various contexts and demonstrates superior performance across zero-shot and few-shot tasks compared to existing models.
Transformer-based multimodal foundation model autoregressive training embeddings text tokens interleaved input sequence unified objective classifying regressing pretraining data sources videos webpages web-scale image-text pairs video-text pairs image-to-text text-to-image tasks in-context image in-context text generation image captioning visual question answering video question answering text-to-image generation multimodal assistants instruction tuning