title: Imp: Highly Capable Large Multimodal Models for Mobile Devices
summary: By harnessing the capabilities of large language models (LLMs), recent large
  multimodal models (LMMs) have shown remarkable versatility in open-world
  multimodal understanding. Nevertheless, they are usually parameter-heavy and
  computation-intensive, thus hindering their applicability in
  resource-constrained scenarios. To this end, several lightweight LMMs have been
  proposed successively to maximize the capabilities under constrained scale
  (e.g., 3B). Despite the encouraging results achieved by these methods, most of
  them only focus on one or two aspects of the design space, and the key design
  choices that influence model capability have not yet been thoroughly
  investigated. In this paper, we conduct a systematic study for lightweight LMMs
  from the aspects of model architecture, training strategy, and training data.
  Based on our findings, we obtain Imp -- a family of highly capable LMMs at the
  2B-4B scales. Notably, our Imp-3B model steadily outperforms all the existing
  lightweight LMMs of similar size, and even surpasses the state-of-the-art LMMs
  at the 13B scale. With low-bit quantization and resolution reduction
  techniques, our Imp model can be deployed on a Qualcomm Snapdragon 8Gen3 mobile
  chip with a high inference speed of about 13 tokens/s.
paper_authors._id: 664c112f877f8871f445dbbc
paper_authors.name: Zhenwei Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T08:37:43.401Z
paper_authors._id: 62d7ce9243df7719860aee18
paper_authors.avatarUrl: /avatars/02d06fe43256a28606c0bb63fed1a441.svg
paper_authors.fullname: Zhenwei Shao
paper_authors.type: user
paper_authors.user: Zenwill
paper_authors._id: 664c112f877f8871f445dbbd
paper_authors.name: Zhou Yu
paper_authors._id: 664c112f877f8871f445dbbe
paper_authors.name: Jun Yu
paper_authors._id: 664c112f877f8871f445dbbf
paper_authors.name: Xuecheng Ouyang
paper_authors._id: 664c112f877f8871f445dbc0
paper_authors.name: Lihao Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T08:38:41.554Z
paper_authors._id: 635dcf0b1e2d8c6673e1255a
paper_authors.avatarUrl: /avatars/174b56f7fe6e0c04de9cf4e387912214.svg
paper_authors.fullname: lihao zheng
paper_authors.type: user
paper_authors.user: NEE2641
paper_authors._id: 664c112f877f8871f445dbc1
paper_authors.name: Zhenbiao Gai
paper_authors._id: 664c112f877f8871f445dbc2
paper_authors.name: Mingyang Wang
paper_authors._id: 664c112f877f8871f445dbc3
paper_authors.name: Jiajun Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T08:39:05.217Z
paper_authors._id: 64c336527f9b6ae10e114d4d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/hlvIpQa4fACIeH9w1hjOZ.jpeg
paper_authors.fullname: Dingjiajun
paper_authors.type: user
paper_authors.user: Kissu4KillU
paper_title: Imp: Highly Capable Large Multimodal Models for Mobile Devices
paper_summary: By harnessing the capabilities of large language models (LLMs), recent large
  multimodal models (LMMs) have shown remarkable versatility in open-world
  multimodal understanding. Nevertheless, they are usually parameter-heavy and
  computation-intensive, thus hindering their applicability in
  resource-constrained scenarios. To this end, several lightweight LMMs have been
  proposed successively to maximize the capabilities under constrained scale
  (e.g., 3B). Despite the encouraging results achieved by these methods, most of
  them only focus on one or two aspects of the design space, and the key design
  choices that influence model capability have not yet been thoroughly
  investigated. In this paper, we conduct a systematic study for lightweight LMMs
  from the aspects of model architecture, training strategy, and training data.
  Based on our findings, we obtain Imp -- a family of highly capable LMMs at the
  2B-4B scales. Notably, our Imp-3B model steadily outperforms all the existing
  lightweight LMMs of similar size, and even surpasses the state-of-the-art LMMs
  at the 13B scale. With low-bit quantization and resolution reduction
  techniques, our Imp model can be deployed on a Qualcomm Snapdragon 8Gen3 mobile
  chip with a high inference speed of about 13 tokens/s.
paper_ai_keywords: large language models (LLMs) large multimodal models (LMMs) open-world multimodal understanding parameter-heavy computation-intensive lightweight LMMs design space model architecture training strategy training data Imp Imp-3B low-bit quantization resolution reduction Qualcomm Snapdragon 8Gen3 mobile chip inference speed