title: TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones
summary: In the era of advanced multimodel learning, multimodal large language models
  (MLLMs) such as GPT-4V have made remarkable strides towards bridging language
  and visual elements. However, the closed-source nature and considerable
  computational demand present notable challenges for universal usage and
  modifications. This is where open-source MLLMs like LLaVA and MiniGPT-4 come
  in, presenting groundbreaking achievements across tasks. Despite these
  accomplishments, computational efficiency remains an unresolved issue, as these
  models, like LLaVA-v1.5-13B, require substantial resources. Addressing these
  issues, we introduce TinyGPT-V, a new-wave model marrying impressive
  performance with commonplace computational capacity. It stands out by requiring
  merely a 24G GPU for training and an 8G GPU or CPU for inference. Built upon
  Phi-2, TinyGPT-V couples an effective language backbone with pre-trained vision
  modules from BLIP-2 or CLIP. TinyGPT-V's 2.8B parameters can undergo a unique
  quantisation process, suitable for local deployment and inference tasks on 8G
  various devices. Our work fosters further developments for designing
  cost-effective, efficient, and high-performing MLLMs, expanding their
  applicability in a broad array of real-world scenarios. Furthermore this paper
  proposed a new paradigm of Multimodal Large Language Model via small backbones.
  Our code and training weights are placed at:
  https://github.com/DLYuanGod/TinyGPT-V and
  https://huggingface.co/Tyrannosaurus/TinyGPT-V respectively.
paper_authors._id: 658e28fdf7291078f975c8f7
paper_authors.name: Zhengqing Yuan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-12-29T02:04:23.827Z
paper_authors._id: 64574d8e182c64e989846ba2
paper_authors.avatarUrl: /avatars/db4bc496a745e1d7de48215b30f6fd3e.svg
paper_authors.fullname: Tyrannosaurus
paper_authors.type: user
paper_authors.user: Tyrannosaurus
paper_authors._id: 658e28fdf7291078f975c8f8
paper_authors.name: Zhaoxu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-04T12:41:23.104Z
paper_authors._id: 64c3c7fd4399efa2fd6a348a
paper_authors.avatarUrl: /avatars/325dccd3ce65205ccc464ada74764683.svg
paper_authors.fullname: LI ZHAOXU
paper_authors.type: user
paper_authors.user: llizhx
paper_authors._id: 658e28fdf7291078f975c8f9
paper_authors.name: Lichao Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T08:29:18.210Z
paper_authors._id: 65a52766215aabac489e3468
paper_authors.avatarUrl: /avatars/fe05e22cd7e12e961296426434e17c76.svg
paper_authors.fullname: Lichao Sun
paper_authors.type: user
paper_authors.user: sunlichao137
paper_title: TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones
paper_summary: In the era of advanced multimodel learning, multimodal large language models
  (MLLMs) such as GPT-4V have made remarkable strides towards bridging language
  and visual elements. However, the closed-source nature and considerable
  computational demand present notable challenges for universal usage and
  modifications. This is where open-source MLLMs like LLaVA and MiniGPT-4 come
  in, presenting groundbreaking achievements across tasks. Despite these
  accomplishments, computational efficiency remains an unresolved issue, as these
  models, like LLaVA-v1.5-13B, require substantial resources. Addressing these
  issues, we introduce TinyGPT-V, a new-wave model marrying impressive
  performance with commonplace computational capacity. It stands out by requiring
  merely a 24G GPU for training and an 8G GPU or CPU for inference. Built upon
  Phi-2, TinyGPT-V couples an effective language backbone with pre-trained vision
  modules from BLIP-2 or CLIP. TinyGPT-V's 2.8B parameters can undergo a unique
  quantisation process, suitable for local deployment and inference tasks on 8G
  various devices. Our work fosters further developments for designing
  cost-effective, efficient, and high-performing MLLMs, expanding their
  applicability in a broad array of real-world scenarios. Furthermore this paper
  proposed a new paradigm of Multimodal Large Language Model via small backbones.
  Our code and training weights are placed at:
  https://github.com/DLYuanGod/TinyGPT-V and
  https://huggingface.co/Tyrannosaurus/TinyGPT-V respectively.
paper_ai_keywords: multimodel learning multimodal large language models (MLLMs) GPT-4V LLaVA MiniGPT-4 TinyGPT-V computational efficiency Phi-2 pre-trained vision modules BLIP-2 CLIP quantisation local deployment Multimodal Large Language Model small backbones