title: MobileLLM: Optimizing Sub-billion Parameter Language Models for
    On-Device Use Cases
summary: This paper addresses the growing need for efficient large language models
  (LLMs) on mobile devices, driven by increasing cloud costs and latency
  concerns. We focus on designing top-quality LLMs with fewer than a billion
  parameters, a practical choice for mobile deployment. Contrary to prevailing
  belief emphasizing the pivotal role of data and parameter quantity in
  determining model quality, our investigation underscores the significance of
  model architecture for sub-billion scale LLMs. Leveraging deep and thin
  architectures, coupled with embedding sharing and grouped-query attention
  mechanisms, we establish a strong baseline network denoted as MobileLLM, which
  attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M
  state-of-the-art models. Additionally, we propose an immediate block-wise
  weight sharing approach with no increase in model size and only marginal
  latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a
  further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover,
  MobileLLM model family shows significant improvements compared to previous
  sub-billion models on chat benchmarks, and demonstrates close correctness to
  LLaMA-v2 7B in API calling tasks, highlighting the capability of small models
  for common on-device use cases.
paper_authors._id: 65dc007f2bc96e6f004a0a4f
paper_authors.name: Zechun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:16:15.652Z
paper_authors._id: 620ae55fe190de4b8f27ba9d
paper_authors.avatarUrl: /avatars/aaa2d0eadbcf96c2eb9059e3d73c2760.svg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: Zechun
paper_authors._id: 65dc007f2bc96e6f004a0a50
paper_authors.name: Changsheng Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:16:22.535Z
paper_authors._id: 64d49ef30f76abaf363b88d6
paper_authors.avatarUrl: /avatars/93b5cc51305ac88198cea1dad8104db2.svg
paper_authors.fullname: Changsheng Zhao
paper_authors.type: user
paper_authors.user: mikezhaocs
paper_authors._id: 65dc007f2bc96e6f004a0a51
paper_authors.name: Forrest Iandola
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:16:29.946Z
paper_authors._id: 5f1753f4925b9863e28ad4de
paper_authors.avatarUrl: /avatars/e56d25a40f2c0e4e6696b9738877fce8.svg
paper_authors.fullname: Forrest Iandola
paper_authors.type: user
paper_authors.user: forresti
paper_authors._id: 65dc007f2bc96e6f004a0a52
paper_authors.name: Chen Lai
paper_authors._id: 65dc007f2bc96e6f004a0a53
paper_authors.name: Yuandong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:16:48.983Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_authors._id: 65dc007f2bc96e6f004a0a54
paper_authors.name: Igor Fedorov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:16:54.965Z
paper_authors._id: 646159d6051604bda02d6f05
paper_authors.avatarUrl: /avatars/d606fee0ad542cc085697b12e673b180.svg
paper_authors.fullname: igor fedorov
paper_authors.type: user
paper_authors.user: igo77
paper_authors._id: 65dc007f2bc96e6f004a0a55
paper_authors.name: Yunyang Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:17:03.222Z
paper_authors._id: 65304b62e7535baecd85d080
paper_authors.avatarUrl: /avatars/6e546c7d1414bd92c5a7c8d8c404de92.svg
paper_authors.fullname: Yunyang Xiong
paper_authors.type: user
paper_authors.user: yunyangx
paper_authors._id: 65dc007f2bc96e6f004a0a56
paper_authors.name: Ernie Chang
paper_authors._id: 65dc007f2bc96e6f004a0a57
paper_authors.name: Yangyang Shi
paper_authors._id: 65dc007f2bc96e6f004a0a58
paper_authors.name: Raghuraman Krishnamoorthi
paper_authors._id: 65dc007f2bc96e6f004a0a59
paper_authors.name: Liangzhen Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:19:03.248Z
paper_authors._id: 64b705c73240387159397c71
paper_authors.avatarUrl: /avatars/da339f07eb58f58756b91eba9af557a1.svg
paper_authors.fullname: Liangzhen Lai
paper_authors.type: user
paper_authors.user: liangzhen-lai
paper_authors._id: 65dc007f2bc96e6f004a0a5a
paper_authors.name: Vikas Chandra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:19:24.981Z
paper_authors._id: 6566681f7b5ed0735812af32
paper_authors.avatarUrl: /avatars/424905abb2973954b0850c592743b6fb.svg
paper_authors.fullname: Vikas Chandra
paper_authors.type: user
paper_authors.user: vchandra
paper_title: MobileLLM: Optimizing Sub-billion Parameter Language Models for
    On-Device Use Cases
paper_summary: This paper addresses the growing need for efficient large language models
  (LLMs) on mobile devices, driven by increasing cloud costs and latency
  concerns. We focus on designing top-quality LLMs with fewer than a billion
  parameters, a practical choice for mobile deployment. Contrary to prevailing
  belief emphasizing the pivotal role of data and parameter quantity in
  determining model quality, our investigation underscores the significance of
  model architecture for sub-billion scale LLMs. Leveraging deep and thin
  architectures, coupled with embedding sharing and grouped-query attention
  mechanisms, we establish a strong baseline network denoted as MobileLLM, which
  attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M
  state-of-the-art models. Additionally, we propose an immediate block-wise
  weight sharing approach with no increase in model size and only marginal
  latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a
  further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover,
  MobileLLM model family shows significant improvements compared to previous
  sub-billion models on chat benchmarks, and demonstrates close correctness to
  LLaMA-v2 7B in API calling tasks, highlighting the capability of small models
  for common on-device use cases.
paper_ai_keywords: deep and thin architectures embedding sharing grouped-query attention mechanisms MobileLLM block-wise weight sharing LLaMA-v2 7B API calling tasks