title: A Survey of Small Language Models
summary: Small Language Models (SLMs) have become increasingly important due to their
  efficiency and performance to perform various language tasks with minimal
  computational resources, making them ideal for various settings including
  on-device, mobile, edge devices, among many others. In this article, we present
  a comprehensive survey on SLMs, focusing on their architectures, training
  techniques, and model compression techniques. We propose a novel taxonomy for
  categorizing the methods used to optimize SLMs, including model compression,
  pruning, and quantization techniques. We summarize the benchmark datasets that
  are useful for benchmarking SLMs along with the evaluation metrics commonly
  used. Additionally, we highlight key open challenges that remain to be
  addressed. Our survey aims to serve as a valuable resource for researchers and
  practitioners interested in developing and deploying small yet efficient
  language models.
paper_authors._id: 672043886e75d23f3e2141db
paper_authors.name: Chien Van Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:54:24.455Z
paper_authors._id: 63edf17df5aef7e657262605
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63edf17df5aef7e657262605/X3WzygIyND_0T_SWRDDU4.jpeg
paper_authors.fullname: Nguyen Van Chien
paper_authors.type: user
paper_authors.user: chiennv
paper_authors._id: 672043886e75d23f3e2141dc
paper_authors.name: Xuan Shen
paper_authors._id: 672043886e75d23f3e2141dd
paper_authors.name: Ryan Aponte
paper_authors._id: 672043886e75d23f3e2141de
paper_authors.name: Yu Xia
paper_authors._id: 672043886e75d23f3e2141df
paper_authors.name: Samyadeep Basu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-31T18:14:25.665Z
paper_authors._id: 6384d891cafe4e4ac4060206
paper_authors.avatarUrl: /avatars/45eb04bf6e0ed1a4daa14070411b693f.svg
paper_authors.fullname: Samyadeep
paper_authors.type: user
paper_authors.user: samyadeepbasu
paper_authors._id: 672043886e75d23f3e2141e0
paper_authors.name: Zhengmian Hu
paper_authors._id: 672043886e75d23f3e2141e1
paper_authors.name: Jian Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:39:18.576Z
paper_authors._id: 64593836c16ecb4815e082ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64593836c16ecb4815e082ce/T9WM0Ly6ADi3a9JUSIsSV.png
paper_authors.fullname: Jian Chen
paper_authors.type: user
paper_authors.user: puar-playground
paper_authors._id: 672043886e75d23f3e2141e2
paper_authors.name: Mihir Parmar
paper_authors._id: 672043886e75d23f3e2141e3
paper_authors.name: Sasidhar Kunapuli
paper_authors._id: 672043886e75d23f3e2141e4
paper_authors.name: Joe Barrow
paper_authors._id: 672043886e75d23f3e2141e5
paper_authors.name: Junda Wu
paper_authors._id: 672043886e75d23f3e2141e6
paper_authors.name: Ashish Singh
paper_authors._id: 672043886e75d23f3e2141e7
paper_authors.name: Yu Wang
paper_authors._id: 672043886e75d23f3e2141e8
paper_authors.name: Jiuxiang Gu
paper_authors._id: 672043886e75d23f3e2141e9
paper_authors.name: Franck Dernoncourt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-31T09:55:11.391Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_authors._id: 672043886e75d23f3e2141ea
paper_authors.name: Nesreen K. Ahmed
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:43:09.275Z
paper_authors._id: 663e588ccca86c8371a913d9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/663e588ccca86c8371a913d9/wgVGw5U77cqwtjovhsXyR.jpeg
paper_authors.fullname: Nesreen Ahmed
paper_authors.type: user
paper_authors.user: nkahmed
paper_authors._id: 672043886e75d23f3e2141eb
paper_authors.name: Nedim Lipka
paper_authors._id: 672043886e75d23f3e2141ec
paper_authors.name: Ruiyi Zhang
paper_authors._id: 672043886e75d23f3e2141ed
paper_authors.name: Xiang Chen
paper_authors._id: 672043886e75d23f3e2141ee
paper_authors.name: Tong Yu
paper_authors._id: 672043886e75d23f3e2141ef
paper_authors.name: Sungchul Kim
paper_authors._id: 672043886e75d23f3e2141f0
paper_authors.name: Hanieh Deilamsalehy
paper_authors._id: 672043886e75d23f3e2141f1
paper_authors.name: Namyong Park
paper_authors._id: 672043886e75d23f3e2141f2
paper_authors.name: Mike Rimer
paper_authors._id: 672043886e75d23f3e2141f3
paper_authors.name: Zhehao Zhang
paper_authors._id: 672043886e75d23f3e2141f4
paper_authors.name: Huanrui Yang
paper_authors._id: 672043886e75d23f3e2141f5
paper_authors.name: Ryan A. Rossi
paper_authors._id: 672043886e75d23f3e2141f6
paper_authors.name: Thien Huu Nguyen
paper_title: A Survey of Small Language Models
paper_summary: Small Language Models (SLMs) have become increasingly important due to their
  efficiency and performance to perform various language tasks with minimal
  computational resources, making them ideal for various settings including
  on-device, mobile, edge devices, among many others. In this article, we present
  a comprehensive survey on SLMs, focusing on their architectures, training
  techniques, and model compression techniques. We propose a novel taxonomy for
  categorizing the methods used to optimize SLMs, including model compression,
  pruning, and quantization techniques. We summarize the benchmark datasets that
  are useful for benchmarking SLMs along with the evaluation metrics commonly
  used. Additionally, we highlight key open challenges that remain to be
  addressed. Our survey aims to serve as a valuable resource for researchers and
  practitioners interested in developing and deploying small yet efficient
  language models.
paper_ai_keywords: Small Language Models (SLMs) model compression pruning quantization