Generative Multimodal Models are In-Context Learners
The human ability to easily solve multimodal tasks in context (i.e., with
only a few demonstrations or simple instructions), is what current multimodal
systems have largely struggled to imitate. In this work, we demonstrate that
the task-agnostic in-context learning capabilities of large multimodal models
can be significantly enhanced by effective scaling-up. We introduce Emu2, a
generative multimodal model with 37 billion parameters, trained on large-scale
multimodal sequences with a unified autoregressive objective. Emu2 exhibits
strong multimodal in-context learning abilities, even emerging to solve tasks
that require on-the-fly reasoning, such as visual prompting and object-grounded
generation. The model sets a new record on multiple multimodal understanding
tasks in few-shot settings. When instruction-tuned to follow specific
instructions, Emu2 further achieves new state-of-the-art on challenging tasks
such as question answering benchmarks for large multimodal models and
open-ended subject-driven generation. These achievements demonstrate that Emu2
can serve as a base model and general-purpose interface for a wide range of
multimodal tasks. Code and models are publicly available to facilitate future
research.
6583a242c73f74776bfe3eae
Quan Sun
admin_assigned
2023-12-21T08:47:07.007Z
630d7a8f81ef9b1772b67f4c
/avatars/00757abd6e548ccebb5bfb233be129a2.svg
Quan Sun
user
QuanSun
6583a242c73f74776bfe3eaf
Yufeng Cui
admin_assigned
2023-12-21T08:47:21.176Z
648683de623b5f050213f2be
/avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
Yufeng Cui
user
YufengCui
6583a242c73f74776bfe3eb0
Xiaosong Zhang
admin_assigned
2023-12-21T08:47:50.401Z
64e473f0b78bc92221ab1883
/avatars/85dd02c2b4879d7696b552f0f9dc2680.svg
Xiaosong Zhang
user
zhangxiaosong18
6583a242c73f74776bfe3eb1
Fan Zhang
claimed_verified
2024-10-08T07:23:04.335Z
640ed40dc025ddf618950af7
/avatars/a99abd4346fa649ad4144f284ebcc972.svg
Fan Zhang
user
ryanzhangfan
6583a242c73f74776bfe3eb2
Qiying Yu
admin_assigned
2023-12-21T08:48:55.715Z
6341375be5071b0c5cc946a3
/avatars/c26a32b54a3f7731854059d247b5c523.svg
Qiying Yu
user
qiying
6583a242c73f74776bfe3eb3
Zhengxiong Luo
claimed_verified
2024-11-25T08:34:57.776Z
64afbb93bbdbfc1b8921960b
/avatars/9ed45fa8a051c10cddfe2c9a3624d619.svg
zhengxiong
user
lzxlog
6583a242c73f74776bfe3eb4
Yueze Wang
claimed_verified
2024-09-27T07:00:10.112Z
6458b59c7a7e192202df8fa0
/avatars/33ee716477e5686da8723d01e199cd27.svg
Yueze Wang
user
yzwang
6583a242c73f74776bfe3eb5
Yongming Rao
admin_assigned
2023-12-21T08:49:21.011Z
63e4865354f51ea342d45d78
/avatars/2e7eccc878751331ca8b282f53e38899.svg
Yongming Rao
user
raoyongming
6583a242c73f74776bfe3eb6
Jingjing Liu
admin_assigned
2023-12-21T08:49:29.009Z
650bf18678d2dfbad85c0db1
/avatars/07ac9d726dc2ccf433746ee858a8044f.svg
jin liu
user
jingjingliu
6583a242c73f74776bfe3eb7
Tiejun Huang
6583a242c73f74776bfe3eb8
Xinlong Wang
admin_assigned
2023-12-21T08:49:44.164Z
63ca558304c979828311c5a5
/avatars/2a439d79fba2f987cabe780d10c94d25.svg
Xinlong Wang
user
xinlongwang
Generative Multimodal Models are In-Context Learners
The human ability to easily solve multimodal tasks in context (i.e., with
only a few demonstrations or simple instructions), is what current multimodal
systems have largely struggled to imitate. In this work, we demonstrate that
the task-agnostic in-context learning capabilities of large multimodal models
can be significantly enhanced by effective scaling-up. We introduce Emu2, a
generative multimodal model with 37 billion parameters, trained on large-scale
multimodal sequences with a unified autoregressive objective. Emu2 exhibits
strong multimodal in-context learning abilities, even emerging to solve tasks
that require on-the-fly reasoning, such as visual prompting and object-grounded
generation. The model sets a new record on multiple multimodal understanding
tasks in few-shot settings. When instruction-tuned to follow specific
instructions, Emu2 further achieves new state-of-the-art on challenging tasks
such as question answering benchmarks for large multimodal models and
open-ended subject-driven generation. These achievements demonstrate that Emu2
can serve as a base model and general-purpose interface for a wide range of
multimodal tasks. Code and models are publicly available to facilitate future
research.
A large-scale generative multimodal model with 37 billion parameters demonstrates strong few-shot in-context learning and achieves state-of-the-art performance on multimodal tasks through scaling-up and instruction tuning.
task-agnostic in-context learning generative multimodal model unified autoregressive objective visual prompting object-grounded generation multimodal understanding tasks few-shot settings question answering benchmarks open-ended subject-driven generation base model general-purpose interface