Multimodal Foundation Models: From Specialists to General-Purpose
  Assistants
This paper presents a comprehensive survey of the taxonomy and evolution of
multimodal foundation models that demonstrate vision and vision-language
capabilities, focusing on the transition from specialist models to
general-purpose assistants. The research landscape encompasses five core
topics, categorized into two classes. (i) We start with a survey of
well-established research areas: multimodal foundation models pre-trained for
specific purposes, including two topics -- methods of learning vision backbones
for visual understanding and text-to-image generation. (ii) Then, we present
recent advances in exploratory, open research areas: multimodal foundation
models that aim to play the role of general-purpose assistants, including three
topics -- unified vision models inspired by large language models (LLMs),
end-to-end training of multimodal LLMs, and chaining multimodal tools with
LLMs. The target audiences of the paper are researchers, graduate students, and
professionals in computer vision and vision-language multimodal communities who
are eager to learn the basics and recent advances in multimodal foundation
models.
650a45c83449d9a49c2b4dae
Chunyuan Li
admin_assigned
2023-09-20T11:30:54.689Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
650a45c83449d9a49c2b4daf
Zhe Gan
admin_assigned
2023-09-20T11:31:02.041Z
644022db21ec7c165d848b6e
/avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
Zhe Gan
user
zhegan27
650a45c83449d9a49c2b4db0
Zhengyuan Yang
admin_assigned
2023-09-20T11:31:10.194Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
650a45c83449d9a49c2b4db1
Jianwei Yang
admin_assigned
2023-09-20T11:31:40.631Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
650a45c83449d9a49c2b4db2
Linjie Li
admin_assigned
2023-09-20T11:31:59.897Z
63db16fff03c3d71ef397206
/avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
Linjie Li
user
linjieli222
650a45c83449d9a49c2b4db3
Lijuan Wang
admin_assigned
2023-09-20T11:32:06.515Z
6413521d4e5305c14f22e110
/avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
Lijuan Wang
user
Lijuan
650a45c83449d9a49c2b4db4
Jianfeng Gao
admin_assigned
2023-09-20T11:32:13.349Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
Multimodal Foundation Models: From Specialists to General-Purpose
  Assistants
This paper presents a comprehensive survey of the taxonomy and evolution of
multimodal foundation models that demonstrate vision and vision-language
capabilities, focusing on the transition from specialist models to
general-purpose assistants. The research landscape encompasses five core
topics, categorized into two classes. (i) We start with a survey of
well-established research areas: multimodal foundation models pre-trained for
specific purposes, including two topics -- methods of learning vision backbones
for visual understanding and text-to-image generation. (ii) Then, we present
recent advances in exploratory, open research areas: multimodal foundation
models that aim to play the role of general-purpose assistants, including three
topics -- unified vision models inspired by large language models (LLMs),
end-to-end training of multimodal LLMs, and chaining multimodal tools with
LLMs. The target audiences of the paper are researchers, graduate students, and
professionals in computer vision and vision-language multimodal communities who
are eager to learn the basics and recent advances in multimodal foundation
models.
The survey covers the development and advancements in multimodal foundation models that integrate vision and language, transitioning from specialized models to versatile assistants through various training methods and unified architectures.
multimodal foundation models vision backbones visual understanding text-to-image generation unified vision models end-to-end training multimodal LLMs chaining multimodal tools large language models (LLMs)