title: Rho-1: Not All Tokens Are What You Need
summary: Previous language model pre-training methods have uniformly applied a
  next-token prediction loss to all training tokens. Challenging this norm, we
  posit that "Not all tokens in a corpus are equally important for language model
  training". Our initial analysis delves into token-level training dynamics of
  language model, revealing distinct loss patterns for different tokens.
  Leveraging these insights, we introduce a new language model called Rho-1.
  Unlike traditional LMs that learn to predict every next token in a corpus,
  Rho-1 employs Selective Language Modeling (SLM), which selectively trains on
  useful tokens that aligned with the desired distribution. This approach
  involves scoring pretraining tokens using a reference model, and then training
  the language model with a focused loss on tokens with higher excess loss. When
  continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute
  improvement in few-shot accuracy of up to 30% in 9 math tasks. After
  fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and
  51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the
  pretraining tokens. Furthermore, when pretraining on 80B general tokens, Rho-1
  achieves 6.8% average enhancement across 15 diverse tasks, increasing both
  efficiency and performance of the language model pre-training.
paper_authors._id: 6618914a507806c4e6796333
paper_authors.name: Zhenghao Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T00:02:49.326Z
paper_authors._id: 63776f1806241efce1e7aae6
paper_authors.avatarUrl: /avatars/d67d9dcd932934c630f407ac152f2ce6.svg
paper_authors.fullname: Zhenghao Lin
paper_authors.type: user
paper_authors.user: Lin0
paper_authors._id: 6618914a507806c4e6796334
paper_authors.name: Zhibin Gou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T04:19:48.766Z
paper_authors._id: 6521678b7bf8cc2dd2a4b02a
paper_authors.avatarUrl: /avatars/3a0aa2f517d8709059a9ecbe83ddfc8a.svg
paper_authors.fullname: Zhibin Gou (alt)
paper_authors.type: user
paper_authors.user: zhibin-msft
paper_authors._id: 6618914a507806c4e6796335
paper_authors.name: Yeyun Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:50:18.623Z
paper_authors._id: 643f615aa16cd6d1f4c581de
paper_authors.avatarUrl: /avatars/47753a3e82b44f81881600c52e1e8495.svg
paper_authors.fullname: Yeyun Gong
paper_authors.type: user
paper_authors.user: yegong
paper_authors._id: 6618914a507806c4e6796336
paper_authors.name: Xiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-15T06:46:08.307Z
paper_authors._id: 63fb6e281b4b1bd4e7ffc5be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: lx865712528
paper_authors._id: 6618914a507806c4e6796337
paper_authors.name: Yelong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:50:25.406Z
paper_authors._id: 6454c337a13edf669cd5d8ea
paper_authors.avatarUrl: /avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
paper_authors.fullname: Yelong Shen
paper_authors.type: user
paper_authors.user: uuu6
paper_authors._id: 6618914a507806c4e6796338
paper_authors.name: Ruochen Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:50:39.473Z
paper_authors._id: 64a234a8d94c3f9ccec8476d
paper_authors.avatarUrl: /avatars/633adf359b1a65f458c3b5a7e59f11c0.svg
paper_authors.fullname: Ruochen Xu
paper_authors.type: user
paper_authors.user: ruochenx
paper_authors._id: 6618914a507806c4e6796339
paper_authors.name: Chen Lin
paper_authors._id: 6618914a507806c4e679633a
paper_authors.name: Yujiu Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:50:56.018Z
paper_authors._id: 64ca1fe838837b12d5e529b7
paper_authors.avatarUrl: /avatars/44a3ad9e59318784ac531993b5f69f6b.svg
paper_authors.fullname: Yujiu Yang
paper_authors.type: user
paper_authors.user: Thu-redrobot
paper_authors._id: 6618914a507806c4e679633b
paper_authors.name: Jian Jiao
paper_authors._id: 6618914a507806c4e679633c
paper_authors.name: Nan Duan
paper_authors._id: 6618914a507806c4e679633d
paper_authors.name: Weizhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:51:22.643Z
paper_authors._id: 64da876370446182be5b608d
paper_authors.avatarUrl: /avatars/e412fdc71404ecdf638e416846e3ebfb.svg
paper_authors.fullname: Weizhu Chen
paper_authors.type: user
paper_authors.user: chenweizhu
paper_title: Rho-1: Not All Tokens Are What You Need
paper_summary: Previous language model pre-training methods have uniformly applied a
  next-token prediction loss to all training tokens. Challenging this norm, we
  posit that "Not all tokens in a corpus are equally important for language model
  training". Our initial analysis delves into token-level training dynamics of
  language model, revealing distinct loss patterns for different tokens.
  Leveraging these insights, we introduce a new language model called Rho-1.
  Unlike traditional LMs that learn to predict every next token in a corpus,
  Rho-1 employs Selective Language Modeling (SLM), which selectively trains on
  useful tokens that aligned with the desired distribution. This approach
  involves scoring pretraining tokens using a reference model, and then training
  the language model with a focused loss on tokens with higher excess loss. When
  continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute
  improvement in few-shot accuracy of up to 30% in 9 math tasks. After
  fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and
  51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the
  pretraining tokens. Furthermore, when pretraining on 80B general tokens, Rho-1
  achieves 6.8% average enhancement across 15 diverse tasks, increasing both
  efficiency and performance of the language model pre-training.
paper_ai_summary: Rho-1, a novel language model using Selective Language Modeling, improves efficiency and performance by selectively training on useful tokens rather than all tokens in the corpus.
paper_ai_keywords: next-token prediction token-level training dynamics Selective Language Modeling SLM reference model excess loss few-shot accuracy MATH dataset DeepSeekMath pretraining tokens efficiency performance