title: Multimodal Foundation Models: From Specialists to General-Purpose
    Assistants
summary: This paper presents a comprehensive survey of the taxonomy and evolution of
  multimodal foundation models that demonstrate vision and vision-language
  capabilities, focusing on the transition from specialist models to
  general-purpose assistants. The research landscape encompasses five core
  topics, categorized into two classes. (i) We start with a survey of
  well-established research areas: multimodal foundation models pre-trained for
  specific purposes, including two topics -- methods of learning vision backbones
  for visual understanding and text-to-image generation. (ii) Then, we present
  recent advances in exploratory, open research areas: multimodal foundation
  models that aim to play the role of general-purpose assistants, including three
  topics -- unified vision models inspired by large language models (LLMs),
  end-to-end training of multimodal LLMs, and chaining multimodal tools with
  LLMs. The target audiences of the paper are researchers, graduate students, and
  professionals in computer vision and vision-language multimodal communities who
  are eager to learn the basics and recent advances in multimodal foundation
  models.
paper_authors._id: 650a45c83449d9a49c2b4dae
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:30:54.689Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 650a45c83449d9a49c2b4daf
paper_authors.name: Zhe Gan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:31:02.041Z
paper_authors._id: 644022db21ec7c165d848b6e
paper_authors.avatarUrl: /avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
paper_authors.fullname: Zhe Gan
paper_authors.type: user
paper_authors.user: zhegan27
paper_authors._id: 650a45c83449d9a49c2b4db0
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:31:10.194Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 650a45c83449d9a49c2b4db1
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:31:40.631Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 650a45c83449d9a49c2b4db2
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:31:59.897Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 650a45c83449d9a49c2b4db3
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:32:06.515Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_authors._id: 650a45c83449d9a49c2b4db4
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:32:13.349Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_title: Multimodal Foundation Models: From Specialists to General-Purpose
    Assistants
paper_summary: This paper presents a comprehensive survey of the taxonomy and evolution of
  multimodal foundation models that demonstrate vision and vision-language
  capabilities, focusing on the transition from specialist models to
  general-purpose assistants. The research landscape encompasses five core
  topics, categorized into two classes. (i) We start with a survey of
  well-established research areas: multimodal foundation models pre-trained for
  specific purposes, including two topics -- methods of learning vision backbones
  for visual understanding and text-to-image generation. (ii) Then, we present
  recent advances in exploratory, open research areas: multimodal foundation
  models that aim to play the role of general-purpose assistants, including three
  topics -- unified vision models inspired by large language models (LLMs),
  end-to-end training of multimodal LLMs, and chaining multimodal tools with
  LLMs. The target audiences of the paper are researchers, graduate students, and
  professionals in computer vision and vision-language multimodal communities who
  are eager to learn the basics and recent advances in multimodal foundation
  models.
paper_ai_keywords: multimodal foundation models vision-language capabilities specialist models general-purpose assistants vision backbones visual understanding text-to-image generation unified vision models large language models (LLMs) end-to-end training multimodal LLMs chaining multimodal tools