title: Generative Multimodal Models are In-Context Learners
summary: The human ability to easily solve multimodal tasks in context (i.e., with
  only a few demonstrations or simple instructions), is what current multimodal
  systems have largely struggled to imitate. In this work, we demonstrate that
  the task-agnostic in-context learning capabilities of large multimodal models
  can be significantly enhanced by effective scaling-up. We introduce Emu2, a
  generative multimodal model with 37 billion parameters, trained on large-scale
  multimodal sequences with a unified autoregressive objective. Emu2 exhibits
  strong multimodal in-context learning abilities, even emerging to solve tasks
  that require on-the-fly reasoning, such as visual prompting and object-grounded
  generation. The model sets a new record on multiple multimodal understanding
  tasks in few-shot settings. When instruction-tuned to follow specific
  instructions, Emu2 further achieves new state-of-the-art on challenging tasks
  such as question answering benchmarks for large multimodal models and
  open-ended subject-driven generation. These achievements demonstrate that Emu2
  can serve as a base model and general-purpose interface for a wide range of
  multimodal tasks. Code and models are publicly available to facilitate future
  research.
paper_authors._id: 6583a242c73f74776bfe3eae
paper_authors.name: Quan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:47:07.007Z
paper_authors._id: 630d7a8f81ef9b1772b67f4c
paper_authors.avatarUrl: /avatars/00757abd6e548ccebb5bfb233be129a2.svg
paper_authors.fullname: Quan Sun
paper_authors.type: user
paper_authors.user: QuanSun
paper_authors._id: 6583a242c73f74776bfe3eaf
paper_authors.name: Yufeng Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:47:21.176Z
paper_authors._id: 648683de623b5f050213f2be
paper_authors.avatarUrl: /avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
paper_authors.fullname: Yufeng Cui
paper_authors.type: user
paper_authors.user: YufengCui
paper_authors._id: 6583a242c73f74776bfe3eb0
paper_authors.name: Xiaosong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:47:50.401Z
paper_authors._id: 64e473f0b78bc92221ab1883
paper_authors.avatarUrl: /avatars/85dd02c2b4879d7696b552f0f9dc2680.svg
paper_authors.fullname: Xiaosong Zhang
paper_authors.type: user
paper_authors.user: zhangxiaosong18
paper_authors._id: 6583a242c73f74776bfe3eb1
paper_authors.name: Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:23:04.335Z
paper_authors._id: 640ed40dc025ddf618950af7
paper_authors.avatarUrl: /avatars/a99abd4346fa649ad4144f284ebcc972.svg
paper_authors.fullname: Fan Zhang
paper_authors.type: user
paper_authors.user: ryanzhangfan
paper_authors._id: 6583a242c73f74776bfe3eb2
paper_authors.name: Qiying Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:48:55.715Z
paper_authors._id: 6341375be5071b0c5cc946a3
paper_authors.avatarUrl: /avatars/c26a32b54a3f7731854059d247b5c523.svg
paper_authors.fullname: Qiying Yu
paper_authors.type: user
paper_authors.user: qiying
paper_authors._id: 6583a242c73f74776bfe3eb3
paper_authors.name: Zhengxiong Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T08:34:57.776Z
paper_authors._id: 64afbb93bbdbfc1b8921960b
paper_authors.avatarUrl: /avatars/9ed45fa8a051c10cddfe2c9a3624d619.svg
paper_authors.fullname: zhengxiong
paper_authors.type: user
paper_authors.user: lzxlog
paper_authors._id: 6583a242c73f74776bfe3eb4
paper_authors.name: Yueze Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-27T07:00:10.112Z
paper_authors._id: 6458b59c7a7e192202df8fa0
paper_authors.avatarUrl: /avatars/33ee716477e5686da8723d01e199cd27.svg
paper_authors.fullname: Yueze Wang
paper_authors.type: user
paper_authors.user: yzwang
paper_authors._id: 6583a242c73f74776bfe3eb5
paper_authors.name: Yongming Rao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:49:21.011Z
paper_authors._id: 63e4865354f51ea342d45d78
paper_authors.avatarUrl: /avatars/2e7eccc878751331ca8b282f53e38899.svg
paper_authors.fullname: Yongming Rao
paper_authors.type: user
paper_authors.user: raoyongming
paper_authors._id: 6583a242c73f74776bfe3eb6
paper_authors.name: Jingjing Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:49:29.009Z
paper_authors._id: 650bf18678d2dfbad85c0db1
paper_authors.avatarUrl: /avatars/07ac9d726dc2ccf433746ee858a8044f.svg
paper_authors.fullname: jin liu
paper_authors.type: user
paper_authors.user: jingjingliu
paper_authors._id: 6583a242c73f74776bfe3eb7
paper_authors.name: Tiejun Huang
paper_authors._id: 6583a242c73f74776bfe3eb8
paper_authors.name: Xinlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:49:44.164Z
paper_authors._id: 63ca558304c979828311c5a5
paper_authors.avatarUrl: /avatars/2a439d79fba2f987cabe780d10c94d25.svg
paper_authors.fullname: Xinlong Wang
paper_authors.type: user
paper_authors.user: xinlongwang
paper_title: Generative Multimodal Models are In-Context Learners
paper_summary: The human ability to easily solve multimodal tasks in context (i.e., with
  only a few demonstrations or simple instructions), is what current multimodal
  systems have largely struggled to imitate. In this work, we demonstrate that
  the task-agnostic in-context learning capabilities of large multimodal models
  can be significantly enhanced by effective scaling-up. We introduce Emu2, a
  generative multimodal model with 37 billion parameters, trained on large-scale
  multimodal sequences with a unified autoregressive objective. Emu2 exhibits
  strong multimodal in-context learning abilities, even emerging to solve tasks
  that require on-the-fly reasoning, such as visual prompting and object-grounded
  generation. The model sets a new record on multiple multimodal understanding
  tasks in few-shot settings. When instruction-tuned to follow specific
  instructions, Emu2 further achieves new state-of-the-art on challenging tasks
  such as question answering benchmarks for large multimodal models and
  open-ended subject-driven generation. These achievements demonstrate that Emu2
  can serve as a base model and general-purpose interface for a wide range of
  multimodal tasks. Code and models are publicly available to facilitate future
  research.
paper_ai_summary: A large-scale generative multimodal model with 37 billion parameters demonstrates strong few-shot in-context learning and achieves state-of-the-art performance on multimodal tasks through scaling-up and instruction tuning.
paper_ai_keywords: task-agnostic in-context learning generative multimodal model unified autoregressive objective visual prompting object-grounded generation multimodal understanding tasks few-shot settings question answering benchmarks open-ended subject-driven generation base model general-purpose interface