title: LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
summary: LLaVA-Plus is a general-purpose multimodal assistant that expands the
  capabilities of large multimodal models. It maintains a skill repository of
  pre-trained vision and vision-language models and can activate relevant tools
  based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on
  multimodal instruction-following data to acquire the ability to use tools,
  covering visual understanding, generation, external knowledge retrieval, and
  compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in
  existing capabilities and exhibits new ones. It is distinct in that the image
  query is directly grounded and actively engaged throughout the entire human-AI
  interaction sessions, significantly improving tool use performance and enabling
  new scenarios.
paper_authors._id: 654db59af5297ada0b9cd352
paper_authors.name: Shilong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T09:06:42.522Z
paper_authors._id: 638b13c0c1d591879698f4e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
paper_authors.fullname: Shilong Liu
paper_authors.type: user
paper_authors.user: ShilongLiu
paper_authors._id: 654db59af5297ada0b9cd353
paper_authors.name: Hao Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-15T09:20:51.063Z
paper_authors._id: 608c49299d7c9519b4adae25
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/608c49299d7c9519b4adae25/pWSW52_QU7VD9hhDUHbmq.jpeg
paper_authors.fullname: Hao Cheng
paper_authors.type: user
paper_authors.user: kelvinih
paper_authors._id: 654db59af5297ada0b9cd354
paper_authors.name: Haotian Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T09:08:46.499Z
paper_authors._id: 63898b61ec1f539adc0f4da2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
paper_authors.fullname: Haotian Liu
paper_authors.type: user
paper_authors.user: liuhaotian
paper_authors._id: 654db59af5297ada0b9cd355
paper_authors.name: Hao Zhang
paper_authors._id: 654db59af5297ada0b9cd356
paper_authors.name: Feng Li
paper_authors._id: 654db59af5297ada0b9cd357
paper_authors.name: Tianhe Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T09:10:15.127Z
paper_authors._id: 6357ba47d3ec8a8aca6ea254
paper_authors.avatarUrl: /avatars/2deb725505aa510a712d91bd0b6b1967.svg
paper_authors.fullname: Ren Tianhe
paper_authors.type: user
paper_authors.user: rentianhe
paper_authors._id: 654db59af5297ada0b9cd358
paper_authors.name: Xueyan Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T09:10:22.150Z
paper_authors._id: 62520988818a5dc29ab91d6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
paper_authors.fullname: Xueyan Zou
paper_authors.type: user
paper_authors.user: xueyanz
paper_authors._id: 654db59af5297ada0b9cd359
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T09:10:51.777Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 654db59af5297ada0b9cd35a
paper_authors.name: Hang Su
paper_authors._id: 654db59af5297ada0b9cd35b
paper_authors.name: Jun Zhu
paper_authors._id: 654db59af5297ada0b9cd35c
paper_authors.name: Lei Zhang
paper_authors._id: 654db59af5297ada0b9cd35d
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T09:13:08.541Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 654db59af5297ada0b9cd35e
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T09:13:16.203Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_title: LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
paper_summary: LLaVA-Plus is a general-purpose multimodal assistant that expands the
  capabilities of large multimodal models. It maintains a skill repository of
  pre-trained vision and vision-language models and can activate relevant tools
  based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on
  multimodal instruction-following data to acquire the ability to use tools,
  covering visual understanding, generation, external knowledge retrieval, and
  compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in
  existing capabilities and exhibits new ones. It is distinct in that the image
  query is directly grounded and actively engaged throughout the entire human-AI
  interaction sessions, significantly improving tool use performance and enabling
  new scenarios.
paper_ai_keywords: multimodal assistant pre-trained vision and vision-language models multimodal instruction-following data visual understanding generation external knowledge retrieval compositions human-AI interaction sessions tool use performance