title: SlimPajama-DC: Understanding Data Combinations for LLM Training
summary: This paper aims to understand the impacts of various data combinations (e.g.,
  web text, wikipedia, github, books) on the training of large language models
  using SlimPajama. SlimPajama is a rigorously deduplicated, multi-source
  dataset, which has been refined and further deduplicated to 627B tokens from
  the extensive 1.2T tokens RedPajama dataset contributed by Together. We've
  termed our research as SlimPajama-DC, an empirical analysis designed to uncover
  fundamental characteristics and best practices associated with employing
  SlimPajama in the training of large language models. During our research with
  SlimPajama, two pivotal observations emerged: (1) Global deduplication vs.
  local deduplication. We analyze and discuss how global (across different
  sources of datasets) and local (within the single source of dataset)
  deduplications affect the performance of trained models. (2) Proportions of
  high-quality/highly-deduplicated multi-source datasets in the combination. To
  study this, we construct six configurations of SlimPajama dataset and train
  individual ones using 1.3B Cerebras-GPT model with Alibi and SwiGLU. Our best
  configuration outperforms the 1.3B model trained on RedPajama using the same
  number of training tokens by a significant margin. All our 1.3B models are
  trained on Cerebras 16times CS-2 cluster with a total of 80 PFLOP/s in bf16
  mixed precision. We further extend our discoveries (such as increasing data
  diversity is crucial after global deduplication) on a 7B model with large
  batch-size training. Our models and the separate SlimPajama-DC datasets are
  available at: https://huggingface.co/MBZUAI-LLM and
  https://huggingface.co/datasets/cerebras/SlimPajama-627B.
paper_authors._id: 650a5199b5a04c8675e3e17f
paper_authors.name: Zhiqiang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:41:00.403Z
paper_authors._id: 649d5a3f4e08d59d20014652
paper_authors.avatarUrl: /avatars/017eccd6aec02137c1beebfe4c951720.svg
paper_authors.fullname: Zhiqiang Shen
paper_authors.type: user
paper_authors.user: Jason0214
paper_authors._id: 650a5199b5a04c8675e3e180
paper_authors.name: Tianhua Tao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:41:08.073Z
paper_authors._id: 61ecb698382b3b92fea7305f
paper_authors.avatarUrl: /avatars/b1bcf0d9ade12172810cb65dc9c4e4d8.svg
paper_authors.fullname: Tianhua Tao
paper_authors.type: user
paper_authors.user: Tianhua
paper_authors._id: 650a5199b5a04c8675e3e181
paper_authors.name: Liqun Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:41:22.407Z
paper_authors._id: 64edc2681d42449431f58c2f
paper_authors.avatarUrl: /avatars/d093ed4b8d5af75225d4393914c85b2a.svg
paper_authors.fullname: Liqun Ma
paper_authors.type: user
paper_authors.user: LiqunMa
paper_authors._id: 650a5199b5a04c8675e3e182
paper_authors.name: Willie Neiswanger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:41:29.279Z
paper_authors._id: 644bf65522d211df6444a7f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644bf65522d211df6444a7f4/k_ddZdQDg2fzhwjI1EXyx.jpeg
paper_authors.fullname: Willie Neiswanger
paper_authors.type: user
paper_authors.user: willieneis
paper_authors._id: 650a5199b5a04c8675e3e183
paper_authors.name: Joel Hestness
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:41:36.254Z
paper_authors._id: 643433124b34368fdb02e8b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643433124b34368fdb02e8b0/mnBzw2JSHk0I5vXGY-YVI.jpeg
paper_authors.fullname: Joel Hestness
paper_authors.type: user
paper_authors.user: jthestness
paper_authors._id: 650a5199b5a04c8675e3e184
paper_authors.name: Natalia Vassilieva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:41:43.211Z
paper_authors._id: 6410072c208821a59b780fcb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6410072c208821a59b780fcb/R9ZDtB9KIaNrLmM2b0GKc.png
paper_authors.fullname: Natalia Vassilieva
paper_authors.type: user
paper_authors.user: vnata
paper_authors._id: 650a5199b5a04c8675e3e185
paper_authors.name: Daria Soboleva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:41:56.885Z
paper_authors._id: 6414bbe095fb6f824b2035a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679080567124-6414bbe095fb6f824b2035a5.jpeg
paper_authors.fullname: Daria Soboleva
paper_authors.type: user
paper_authors.user: daria-soboleva
paper_authors._id: 650a5199b5a04c8675e3e186
paper_authors.name: Eric Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:42:05.934Z
paper_authors._id: 64ff67722ad36636be6c4542
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
paper_authors.fullname: Eric Xing
paper_authors.type: user
paper_authors.user: EricX003
paper_title: SlimPajama-DC: Understanding Data Combinations for LLM Training
paper_summary: This paper aims to understand the impacts of various data combinations (e.g.,
  web text, wikipedia, github, books) on the training of large language models
  using SlimPajama. SlimPajama is a rigorously deduplicated, multi-source
  dataset, which has been refined and further deduplicated to 627B tokens from
  the extensive 1.2T tokens RedPajama dataset contributed by Together. We've
  termed our research as SlimPajama-DC, an empirical analysis designed to uncover
  fundamental characteristics and best practices associated with employing
  SlimPajama in the training of large language models. During our research with
  SlimPajama, two pivotal observations emerged: (1) Global deduplication vs.
  local deduplication. We analyze and discuss how global (across different
  sources of datasets) and local (within the single source of dataset)
  deduplications affect the performance of trained models. (2) Proportions of
  high-quality/highly-deduplicated multi-source datasets in the combination. To
  study this, we construct six configurations of SlimPajama dataset and train
  individual ones using 1.3B Cerebras-GPT model with Alibi and SwiGLU. Our best
  configuration outperforms the 1.3B model trained on RedPajama using the same
  number of training tokens by a significant margin. All our 1.3B models are
  trained on Cerebras 16times CS-2 cluster with a total of 80 PFLOP/s in bf16
  mixed precision. We further extend our discoveries (such as increasing data
  diversity is crucial after global deduplication) on a 7B model with large
  batch-size training. Our models and the separate SlimPajama-DC datasets are
  available at: https://huggingface.co/MBZUAI-LLM and
  https://huggingface.co/datasets/cerebras/SlimPajama-627B.
paper_ai_keywords: SlimPajama RedPajama SlimPajama-DC Global deduplication Local deduplication Cerebras-GPT Alibi SwiGLU Cerebras 16$\times$ CS-2 cluster bf16 mixed precision Large batch-size training