LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
LLaVA-Plus is a general-purpose multimodal assistant that expands the
capabilities of large multimodal models. It maintains a skill repository of
pre-trained vision and vision-language models and can activate relevant tools
based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on
multimodal instruction-following data to acquire the ability to use tools,
covering visual understanding, generation, external knowledge retrieval, and
compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in
existing capabilities and exhibits new ones. It is distinct in that the image
query is directly grounded and actively engaged throughout the entire human-AI
interaction sessions, significantly improving tool use performance and enabling
new scenarios.
654db59af5297ada0b9cd352
Shilong Liu
admin_assigned
2023-11-10T09:06:42.522Z
638b13c0c1d591879698f4e2
https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
Shilong Liu
user
ShilongLiu
654db59af5297ada0b9cd353
Hao Cheng
claimed_verified
2023-11-15T09:20:51.063Z
608c49299d7c9519b4adae25
https://cdn-avatars.huggingface.co/v1/production/uploads/608c49299d7c9519b4adae25/pWSW52_QU7VD9hhDUHbmq.jpeg
Hao Cheng
user
kelvinih
654db59af5297ada0b9cd354
Haotian Liu
admin_assigned
2023-11-10T09:08:46.499Z
63898b61ec1f539adc0f4da2
https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
Haotian Liu
user
liuhaotian
654db59af5297ada0b9cd355
Hao Zhang
654db59af5297ada0b9cd356
Feng Li
654db59af5297ada0b9cd357
Tianhe Ren
admin_assigned
2023-11-10T09:10:15.127Z
6357ba47d3ec8a8aca6ea254
/avatars/2deb725505aa510a712d91bd0b6b1967.svg
Ren Tianhe
user
rentianhe
654db59af5297ada0b9cd358
Xueyan Zou
admin_assigned
2023-11-10T09:10:22.150Z
62520988818a5dc29ab91d6f
https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
Xueyan Zou
user
xueyanz
654db59af5297ada0b9cd359
Jianwei Yang
admin_assigned
2023-11-10T09:10:51.777Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
654db59af5297ada0b9cd35a
Hang Su
654db59af5297ada0b9cd35b
Jun Zhu
654db59af5297ada0b9cd35c
Lei Zhang
654db59af5297ada0b9cd35d
Jianfeng Gao
admin_assigned
2023-11-10T09:13:08.541Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
654db59af5297ada0b9cd35e
Chunyuan Li
admin_assigned
2023-11-10T09:13:16.203Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
LLaVA-Plus is a general-purpose multimodal assistant that expands the
capabilities of large multimodal models. It maintains a skill repository of
pre-trained vision and vision-language models and can activate relevant tools
based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on
multimodal instruction-following data to acquire the ability to use tools,
covering visual understanding, generation, external knowledge retrieval, and
compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in
existing capabilities and exhibits new ones. It is distinct in that the image
query is directly grounded and actively engaged throughout the entire human-AI
interaction sessions, significantly improving tool use performance and enabling
new scenarios.
LLaVA-Plus, a general-purpose multimodal assistant, enhances large multimodal models by integrating pre-trained vision and vision-language models, performing tool-assisted tasks and improving interaction through direct image grounding.
multimodal assistant pre-trained vision and vision-language models multimodal instruction-following data tool use visual understanding generation external knowledge retrieval compositions human-AI interaction direct image grounding