title: SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language
    Model
summary: While large language models have facilitated breakthroughs in many
  applications of artificial intelligence, their inherent largeness makes them
  computationally expensive and challenging to deploy in resource-constrained
  settings. In this paper, we document the development of SmolLM2, a
  state-of-the-art "small" (1.7 billion parameter) language model (LM). To attain
  strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a
  multi-stage training process that mixes web text with specialized math, code,
  and instruction-following data. We additionally introduce new specialized
  datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing
  datasets to be problematically small or low-quality. To inform our design
  decisions, we perform both small-scale ablations as well as a manual refinement
  process that updates the dataset mixing rates at each stage based on the
  performance at the previous stage. Ultimately, we demonstrate that SmolLM2
  outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To
  facilitate future research on LM development as well as applications of small
  LMs, we release both SmolLM2 as well as all of the datasets we prepared in the
  course of this project.
paper_authors._id: 67a446a9430e358f5d5ac4c3
paper_authors.name: Loubna Ben Allal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:14:56.506Z
paper_authors._id: 61c141342aac764ce1654e43
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61c141342aac764ce1654e43/81AwoT5IQ_Xdw0OVw7TKu.jpeg
paper_authors.fullname: Loubna Ben Allal
paper_authors.type: user
paper_authors.user: loubnabnl
paper_authors._id: 67a446a9430e358f5d5ac4c4
paper_authors.name: Anton Lozhkov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:14:39.237Z
paper_authors._id: 602e6dee60e3dd96631c906e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1613655355830-noauth.png
paper_authors.fullname: Anton Lozhkov
paper_authors.type: user
paper_authors.user: anton-l
paper_authors._id: 67a446a9430e358f5d5ac4c5
paper_authors.name: Elie Bakouch
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:14:45.734Z
paper_authors._id: 651e96991b97c9f33d26bde6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651e96991b97c9f33d26bde6/-Bqs6qrmz0yCfwtB2e-6q.jpeg
paper_authors.fullname: Elie Bakouch
paper_authors.type: user
paper_authors.user: eliebak
paper_authors._id: 67a446a9430e358f5d5ac4c6
paper_authors.name: Gabriel Martín Blázquez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:14:43.746Z
paper_authors._id: 60f2fc91b92afccb7c34b8ed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60f2fc91b92afccb7c34b8ed/W2-Nay12Ef4Ltyaf8EKE9.jpeg
paper_authors.fullname: Gabriel Martín Blázquez
paper_authors.type: user
paper_authors.user: gabrielmbmb
paper_authors._id: 67a446a9430e358f5d5ac4c7
paper_authors.name: Guilherme Penedo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:14:50.085Z
paper_authors._id: 62596f9e1c0a084224b93e00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62596f9e1c0a084224b93e00/X2aLkJ0ofhkXwAg7lXvxD.jpeg
paper_authors.fullname: Guilherme Penedo
paper_authors.type: user
paper_authors.user: guipenedo
paper_authors._id: 67a446a9430e358f5d5ac4c8
paper_authors.name: Lewis Tunstall
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:16:30.456Z
paper_authors._id: 5f0c746619cb630495b814fd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg
paper_authors.fullname: Lewis Tunstall
paper_authors.type: user
paper_authors.user: lewtun
paper_authors._id: 67a446a9430e358f5d5ac4c9
paper_authors.name: Andrés Marafioti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:16:37.742Z
paper_authors._id: 65d66b494bbd0d92b641cdbb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg
paper_authors.fullname: Andres Marafioti
paper_authors.type: user
paper_authors.user: andito
paper_authors._id: 67a446a9430e358f5d5ac4ca
paper_authors.name: Hynek Kydlíček
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:16:43.590Z
paper_authors._id: 626ede24d2fa9e7d598c8709
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626ede24d2fa9e7d598c8709/JKS8-Y2Jw87EgNQZBRswq.jpeg
paper_authors.fullname: Hynek Kydlicek
paper_authors.type: user
paper_authors.user: hynky
paper_authors._id: 67a446a9430e358f5d5ac4cb
paper_authors.name: Agustín Piqueres Lajarín
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:16:49.324Z
paper_authors._id: 6435d564a4bd75c62cc03701
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6435d564a4bd75c62cc03701/7P2G_wVNB6MISp2Phh427.jpeg
paper_authors.fullname: Agustín Piqueres Lajarín
paper_authors.type: user
paper_authors.user: plaguss
paper_authors._id: 67a446a9430e358f5d5ac4cc
paper_authors.name: Vaibhav Srivastav
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:14:52.239Z
paper_authors._id: 61b85ce86eb1f2c5e6233736
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1655385361868-61b85ce86eb1f2c5e6233736.jpeg
paper_authors.fullname: Vaibhav Srivastav
paper_authors.type: user
paper_authors.user: reach-vb
paper_authors._id: 67a446a9430e358f5d5ac4cd
paper_authors.name: Joshua Lochner
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:14:36.878Z
paper_authors._id: 61b253b7ac5ecaae3d1efe0c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61b253b7ac5ecaae3d1efe0c/hwiQ0uvz3t-L5a-NtBIO6.png
paper_authors.fullname: Joshua
paper_authors.type: user
paper_authors.user: Xenova
paper_authors._id: 67a446a9430e358f5d5ac4ce
paper_authors.name: Caleb Fahlgren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:16:56.849Z
paper_authors._id: 648a374f00f7a3374ee64b99
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648a374f00f7a3374ee64b99/YPwSOrronoozwHbJchPn3.jpeg
paper_authors.fullname: Caleb Fahlgren
paper_authors.type: user
paper_authors.user: cfahlgren1
paper_authors._id: 67a446a9430e358f5d5ac4cf
paper_authors.name: Xuan-Son Nguyen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:17:02.477Z
paper_authors._id: 63ca214abedad7e2bf1d1517
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674191139776-noauth.png
paper_authors.fullname: Xuan-Son Nguyen
paper_authors.type: user
paper_authors.user: ngxson
paper_authors._id: 67a446a9430e358f5d5ac4d0
paper_authors.name: Clémentine Fourrier
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:14:54.591Z
paper_authors._id: 6202a599216215a22221dea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png
paper_authors.fullname: Clémentine Fourrier
paper_authors.type: user
paper_authors.user: clefourrier
paper_authors._id: 67a446a9430e358f5d5ac4d1
paper_authors.name: Ben Burtenshaw
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:14:41.918Z
paper_authors._id: 62d648291fa3e4e7ae3fa6e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d648291fa3e4e7ae3fa6e8/oatOwf8Xqe5eDbCSuYqCd.png
paper_authors.fullname: ben burtenshaw
paper_authors.type: user
paper_authors.user: burtenshaw
paper_authors._id: 67a446a9430e358f5d5ac4d2
paper_authors.name: Hugo Larcher
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:17:14.767Z
paper_authors._id: 641cc77c92cd25302998b740
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641cc77c92cd25302998b740/5A81W5s3ecLaLXFir52Rw.jpeg
paper_authors.fullname: Hugo Larcher
paper_authors.type: user
paper_authors.user: hlarcher
paper_authors._id: 67a446a9430e358f5d5ac4d3
paper_authors.name: Haojun Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:17:33.798Z
paper_authors._id: 660ed80b1889bf2cd53cab7f
paper_authors.avatarUrl: /avatars/93ee6ff00668c2698ad8b6fa6f072b92.svg
paper_authors.fullname: Haojun Zhao
paper_authors.type: user
paper_authors.user: zzhhjjj
paper_authors._id: 67a446a9430e358f5d5ac4d4
paper_authors.name: Cyril Zakka
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:17:43.679Z
paper_authors._id: 66ba71a4447411b9c0e19d71
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4f93ZrYdaKfK3F53IB51x.jpeg
paper_authors.fullname: Cyril
paper_authors.type: user
paper_authors.user: cyrilzakka
paper_authors._id: 67a446a9430e358f5d5ac4d5
paper_authors.name: Mathieu Morlon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:17:50.199Z
paper_authors._id: 664d7d1e4f54c9372970e121
paper_authors.avatarUrl: /avatars/695a209d6951a4623eceedcd2eed3a68.svg
paper_authors.fullname: Mathieu Morlon
paper_authors.type: user
paper_authors.user: glutamatt
paper_authors._id: 67a446a9430e358f5d5ac4d6
paper_authors.name: Colin Raffel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:17:57.936Z
paper_authors._id: 6079c29765b9d0165cb18392
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
paper_authors.fullname: Colin Raffel
paper_authors.type: user
paper_authors.user: craffel
paper_authors._id: 67a446a9430e358f5d5ac4d7
paper_authors.name: Leandro von Werra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T11:03:02.572Z
paper_authors._id: 5e48005437cb5b49818287a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
paper_authors.fullname: Leandro von Werra
paper_authors.type: user
paper_authors.user: lvwerra
paper_authors._id: 67a446a9430e358f5d5ac4d8
paper_authors.name: Thomas Wolf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-06T14:14:14.159Z
paper_authors._id: 5df7e9e5da6d0311fd3d53f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
paper_authors.fullname: Thomas Wolf
paper_authors.type: user
paper_authors.user: thomwolf
paper_title: SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language
    Model
paper_summary: While large language models have facilitated breakthroughs in many
  applications of artificial intelligence, their inherent largeness makes them
  computationally expensive and challenging to deploy in resource-constrained
  settings. In this paper, we document the development of SmolLM2, a
  state-of-the-art "small" (1.7 billion parameter) language model (LM). To attain
  strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a
  multi-stage training process that mixes web text with specialized math, code,
  and instruction-following data. We additionally introduce new specialized
  datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing
  datasets to be problematically small or low-quality. To inform our design
  decisions, we perform both small-scale ablations as well as a manual refinement
  process that updates the dataset mixing rates at each stage based on the
  performance at the previous stage. Ultimately, we demonstrate that SmolLM2
  outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To
  facilitate future research on LM development as well as applications of small
  LMs, we release both SmolLM2 as well as all of the datasets we prepared in the
  course of this project.
paper_ai_keywords: Contrast language models computational expensive deploy SmolLM2 multi-stage training web text specialized math instruction-following data FineMath Stack-Edu SmolTalk small-scale ablations manual refinement Qwen2.5-1.5B Llama3.2-1B small LMs