title: Multimodal Foundation Models: From Specialists to General-Purpose
    Assistants
summary: This paper presents a comprehensive survey of the taxonomy and evolution of
  multimodal foundation models that demonstrate vision and vision-language
  capabilities, focusing on the transition from specialist models to
  general-purpose assistants. The research landscape encompasses five core
  topics, categorized into two classes. (i) We start with a survey of
  well-established research areas: multimodal foundation models pre-trained for
  specific purposes, including two topics -- methods of learning vision backbones
  for visual understanding and text-to-image generation. (ii) Then, we present
  recent advances in exploratory, open research areas: multimodal foundation
  models that aim to play the role of general-purpose assistants, including three
  topics -- unified vision models inspired by large language models (LLMs),
  end-to-end training of multimodal LLMs, and chaining multimodal tools with
  LLMs. The target audiences of the paper are researchers, graduate students, and
  professionals in computer vision and vision-language multimodal communities who
  are eager to learn the basics and recent advances in multimodal foundation
  models.
paper_authors._id: 650a45c83449d9a49c2b4dae
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:30:54.689Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 650a45c83449d9a49c2b4daf
paper_authors.name: Zhe Gan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:31:02.041Z
paper_authors._id: 644022db21ec7c165d848b6e
paper_authors.avatarUrl: /avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
paper_authors.fullname: Zhe Gan
paper_authors.type: user
paper_authors.user: zhegan27
paper_authors._id: 650a45c83449d9a49c2b4db0
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:31:10.194Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 650a45c83449d9a49c2b4db1
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:31:40.631Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 650a45c83449d9a49c2b4db2
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:31:59.897Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 650a45c83449d9a49c2b4db3
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:32:06.515Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_authors._id: 650a45c83449d9a49c2b4db4
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:32:13.349Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_title: Multimodal Foundation Models: From Specialists to General-Purpose
    Assistants
paper_summary: This paper presents a comprehensive survey of the taxonomy and evolution of
  multimodal foundation models that demonstrate vision and vision-language
  capabilities, focusing on the transition from specialist models to
  general-purpose assistants. The research landscape encompasses five core
  topics, categorized into two classes. (i) We start with a survey of
  well-established research areas: multimodal foundation models pre-trained for
  specific purposes, including two topics -- methods of learning vision backbones
  for visual understanding and text-to-image generation. (ii) Then, we present
  recent advances in exploratory, open research areas: multimodal foundation
  models that aim to play the role of general-purpose assistants, including three
  topics -- unified vision models inspired by large language models (LLMs),
  end-to-end training of multimodal LLMs, and chaining multimodal tools with
  LLMs. The target audiences of the paper are researchers, graduate students, and
  professionals in computer vision and vision-language multimodal communities who
  are eager to learn the basics and recent advances in multimodal foundation
  models.
paper_ai_summary: The survey covers the development and advancements in multimodal foundation models that integrate vision and language, transitioning from specialized models to versatile assistants through various training methods and unified architectures.
paper_ai_keywords: multimodal foundation models vision backbones visual understanding text-to-image generation unified vision models end-to-end training multimodal LLMs chaining multimodal tools large language models (LLMs)