SlimPajama-DC: Understanding Data Combinations for LLM Training
This paper aims to understand the impacts of various data combinations (e.g.,
web text, wikipedia, github, books) on the training of large language models
using SlimPajama. SlimPajama is a rigorously deduplicated, multi-source
dataset, which has been refined and further deduplicated to 627B tokens from
the extensive 1.2T tokens RedPajama dataset contributed by Together. We've
termed our research as SlimPajama-DC, an empirical analysis designed to uncover
fundamental characteristics and best practices associated with employing
SlimPajama in the training of large language models. During our research with
SlimPajama, two pivotal observations emerged: (1) Global deduplication vs.
local deduplication. We analyze and discuss how global (across different
sources of datasets) and local (within the single source of dataset)
deduplications affect the performance of trained models. (2) Proportions of
high-quality/highly-deduplicated multi-source datasets in the combination. To
study this, we construct six configurations of SlimPajama dataset and train
individual ones using 1.3B Cerebras-GPT model with Alibi and SwiGLU. Our best
configuration outperforms the 1.3B model trained on RedPajama using the same
number of training tokens by a significant margin. All our 1.3B models are
trained on Cerebras 16times CS-2 cluster with a total of 80 PFLOP/s in bf16
mixed precision. We further extend our discoveries (such as increasing data
diversity is crucial after global deduplication) on a 7B model with large
batch-size training. Our models and the separate SlimPajama-DC datasets are
available at: https://huggingface.co/MBZUAI-LLM and
https://huggingface.co/datasets/cerebras/SlimPajama-627B.
650a5199b5a04c8675e3e17f
Zhiqiang Shen
admin_assigned
2023-09-20T11:41:00.403Z
649d5a3f4e08d59d20014652
/avatars/017eccd6aec02137c1beebfe4c951720.svg
Zhiqiang Shen
user
Jason0214
650a5199b5a04c8675e3e180
Tianhua Tao
admin_assigned
2023-09-20T11:41:08.073Z
61ecb698382b3b92fea7305f
/avatars/b1bcf0d9ade12172810cb65dc9c4e4d8.svg
Tianhua Tao
user
Tianhua
650a5199b5a04c8675e3e181
Liqun Ma
admin_assigned
2023-09-20T11:41:22.407Z
64edc2681d42449431f58c2f
https://cdn-avatars.huggingface.co/v1/production/uploads/64edc2681d42449431f58c2f/kWG8iyO-lSMt-2Vbm9ipq.jpeg
Liqun Ma
user
LiqunMa
650a5199b5a04c8675e3e182
Willie Neiswanger
admin_assigned
2023-09-20T11:41:29.279Z
644bf65522d211df6444a7f4
https://cdn-avatars.huggingface.co/v1/production/uploads/644bf65522d211df6444a7f4/k_ddZdQDg2fzhwjI1EXyx.jpeg
Willie Neiswanger
user
willieneis
650a5199b5a04c8675e3e183
Joel Hestness
admin_assigned
2023-09-20T11:41:36.254Z
643433124b34368fdb02e8b0
https://cdn-avatars.huggingface.co/v1/production/uploads/643433124b34368fdb02e8b0/mnBzw2JSHk0I5vXGY-YVI.jpeg
Joel Hestness
user
jthestness
650a5199b5a04c8675e3e184
Natalia Vassilieva
admin_assigned
2023-09-20T11:41:43.211Z
6410072c208821a59b780fcb
https://cdn-avatars.huggingface.co/v1/production/uploads/6410072c208821a59b780fcb/R9ZDtB9KIaNrLmM2b0GKc.png
Natalia Vassilieva
user
vnata
650a5199b5a04c8675e3e185
Daria Soboleva
admin_assigned
2023-09-20T11:41:56.885Z
6414bbe095fb6f824b2035a5
https://cdn-avatars.huggingface.co/v1/production/uploads/1679080567124-6414bbe095fb6f824b2035a5.jpeg
Daria Soboleva
user
daria-soboleva
650a5199b5a04c8675e3e186
Eric Xing
admin_assigned
2023-09-20T11:42:05.934Z
64ff67722ad36636be6c4542
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
Eric Xing
user
EricX003
SlimPajama-DC: Understanding Data Combinations for LLM Training
This paper aims to understand the impacts of various data combinations (e.g.,
web text, wikipedia, github, books) on the training of large language models
using SlimPajama. SlimPajama is a rigorously deduplicated, multi-source
dataset, which has been refined and further deduplicated to 627B tokens from
the extensive 1.2T tokens RedPajama dataset contributed by Together. We've
termed our research as SlimPajama-DC, an empirical analysis designed to uncover
fundamental characteristics and best practices associated with employing
SlimPajama in the training of large language models. During our research with
SlimPajama, two pivotal observations emerged: (1) Global deduplication vs.
local deduplication. We analyze and discuss how global (across different
sources of datasets) and local (within the single source of dataset)
deduplications affect the performance of trained models. (2) Proportions of
high-quality/highly-deduplicated multi-source datasets in the combination. To
study this, we construct six configurations of SlimPajama dataset and train
individual ones using 1.3B Cerebras-GPT model with Alibi and SwiGLU. Our best
configuration outperforms the 1.3B model trained on RedPajama using the same
number of training tokens by a significant margin. All our 1.3B models are
trained on Cerebras 16times CS-2 cluster with a total of 80 PFLOP/s in bf16
mixed precision. We further extend our discoveries (such as increasing data
diversity is crucial after global deduplication) on a 7B model with large
batch-size training. Our models and the separate SlimPajama-DC datasets are
available at: https://huggingface.co/MBZUAI-LLM and
https://huggingface.co/datasets/cerebras/SlimPajama-627B.
Research on SlimPajama-DC reveals the impact of various data deduplication strategies and proportions on large language model training, showing improved performance with optimal data diversity.
deduplication SlimPajama RedPajama Cerebras-GPT Alibi SwiGLU Cerebras 16$\times$ CS-2 cluster bf16 mixed precision