title: MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT
summary: "Bigger the better" has been the predominant trend in recent Large Language
  Models (LLMs) development. However, LLMs do not suit well for scenarios that
  require on-device processing, energy efficiency, low memory footprint, and
  response efficiency. These requisites are crucial for privacy, security, and
  sustainable deployment. This paper explores the "less is more" paradigm by
  addressing the challenge of designing accurate yet efficient Small Language
  Models (SLMs) for resource constrained devices. Our primary contribution is the
  introduction of an accurate and fully transparent open-source 0.5 billion
  (0.5B) parameter SLM, named MobiLlama, catering to the specific needs of
  resource-constrained computing with an emphasis on enhanced performance with
  reduced resource demands. MobiLlama is a SLM design that initiates from a
  larger model and applies a careful parameter sharing scheme to reduce both the
  pre-training and the deployment cost. Our work strives to not only bridge the
  gap in open-source SLMs but also ensures full transparency, where complete
  training data pipeline, training code, model weights, and over 300 checkpoints
  along with evaluation codes is available at :
  https://github.com/mbzuai-oryx/MobiLlama.
paper_authors._id: 65dd76e84a4fce1ec96347f4
paper_authors.name: Omkar Thawakar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:29:58.983Z
paper_authors._id: 64b7d2ad8c632fbca9507431
paper_authors.avatarUrl: /avatars/76c31ea218108cf6c3715269f7605404.svg
paper_authors.fullname: Omkar Thawakar
paper_authors.type: user
paper_authors.user: omkarthawakar
paper_authors._id: 65dd76e84a4fce1ec96347f5
paper_authors.name: Ashmal Vayani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:30:14.795Z
paper_authors._id: 64f9d0ded88459d40bd0efb9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f9d0ded88459d40bd0efb9/16Y41IUX0T_TZj_g_tTBF.jpeg
paper_authors.fullname: Ashmal Vayani
paper_authors.type: user
paper_authors.user: Ashmal
paper_authors._id: 65dd76e84a4fce1ec96347f6
paper_authors.name: Salman Khan
paper_authors._id: 65dd76e84a4fce1ec96347f7
paper_authors.name: Hisham Cholakal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:30:47.610Z
paper_authors._id: 654a5f4f9b8bd6406d45bb46
paper_authors.avatarUrl: /avatars/ac0d7eef62cd98a280b162cf7896b1a2.svg
paper_authors.fullname: Hisham Cholakkal
paper_authors.type: user
paper_authors.user: hishamcholakkal
paper_authors._id: 65dd76e84a4fce1ec96347f8
paper_authors.name: Rao M. Anwer
paper_authors._id: 65dd76e84a4fce1ec96347f9
paper_authors.name: Michael Felsberg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:31:08.550Z
paper_authors._id: 63d38a1a9e1b81818c816905
paper_authors.avatarUrl: /avatars/a2160bb9fbb8a8eec129b8eab559294b.svg
paper_authors.fullname: Michael Felsberg
paper_authors.type: user
paper_authors.user: CVL-LiU
paper_authors._id: 65dd76e84a4fce1ec96347fa
paper_authors.name: Tim Baldwin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:31:14.438Z
paper_authors._id: 65d895d8d435dce78392065f
paper_authors.avatarUrl: /avatars/68c7185d3dd1adf78927b91849978d62.svg
paper_authors.fullname: Tim Baldwin
paper_authors.type: user
paper_authors.user: tbaldwin
paper_authors._id: 65dd76e84a4fce1ec96347fb
paper_authors.name: Eric P. Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:31:25.777Z
paper_authors._id: 64ff67722ad36636be6c4542
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
paper_authors.fullname: Eric Xing
paper_authors.type: user
paper_authors.user: EricX003
paper_authors._id: 65dd76e84a4fce1ec96347fc
paper_authors.name: Fahad Shahbaz Khan
paper_title: MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT
paper_summary: "Bigger the better" has been the predominant trend in recent Large Language
  Models (LLMs) development. However, LLMs do not suit well for scenarios that
  require on-device processing, energy efficiency, low memory footprint, and
  response efficiency. These requisites are crucial for privacy, security, and
  sustainable deployment. This paper explores the "less is more" paradigm by
  addressing the challenge of designing accurate yet efficient Small Language
  Models (SLMs) for resource constrained devices. Our primary contribution is the
  introduction of an accurate and fully transparent open-source 0.5 billion
  (0.5B) parameter SLM, named MobiLlama, catering to the specific needs of
  resource-constrained computing with an emphasis on enhanced performance with
  reduced resource demands. MobiLlama is a SLM design that initiates from a
  larger model and applies a careful parameter sharing scheme to reduce both the
  pre-training and the deployment cost. Our work strives to not only bridge the
  gap in open-source SLMs but also ensures full transparency, where complete
  training data pipeline, training code, model weights, and over 300 checkpoints
  along with evaluation codes is available at :
  https://github.com/mbzuai-oryx/MobiLlama.
paper_ai_summary: MobiLlama is a 0.5 billion parameter small language model designed for resource-constrained devices using parameter sharing to balance accuracy and efficiency.
paper_ai_keywords: Large Language Models Small Language Models resource constrained devices parameter sharing MobiLlama pre-training deployment cost full transparency training data pipeline training code model weights checkpoints evaluation codes