DataComp-LM: In search of the next generation of training sets for
  language models
We introduce DataComp for Language Models (DCLM), a testbed for controlled
dataset experiments with the goal of improving language models. As part of
DCLM, we provide a standardized corpus of 240T tokens extracted from Common
Crawl, effective pretraining recipes based on the OpenLM framework, and a broad
suite of 53 downstream evaluations. Participants in the DCLM benchmark can
experiment with data curation strategies such as deduplication, filtering, and
data mixing at model scales ranging from 412M to 7B parameters. As a baseline
for DCLM, we conduct extensive experiments and find that model-based filtering
is key to assembling a high-quality training set. The resulting dataset,
DCLM-Baseline enables training a 7B parameter language model from scratch to
64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the
previous state-of-the-art in open-data language models, DCLM-Baseline
represents a 6.6 percentage point improvement on MMLU while being trained with
40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and
Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53
natural language understanding tasks while being trained with 6.6x less compute
than Llama 3 8B. Our results highlight the importance of dataset design for
training language models and offer a starting point for further research on
data curation.
6670f297d5e1408bcf417f1a
Jeffrey Li
admin_assigned
2024-06-18T08:32:22.843Z
63ca51a82ffd350b485f9fa5
https://cdn-avatars.huggingface.co/v1/production/uploads/1674203513156-noauth.jpeg
Jeffrey Li
user
JeffreyLi
6670f297d5e1408bcf417f1b
Alex Fang
6670f297d5e1408bcf417f1c
Georgios Smyrnis
6670f297d5e1408bcf417f1d
Maor Ivgi
claimed_verified
2024-07-10T08:20:07.624Z
610b7f53b1d06c607ab4436d
https://cdn-avatars.huggingface.co/v1/production/uploads/610b7f53b1d06c607ab4436d/Bqw7fIsjpuukFP11S-awA.jpeg
Maor
user
Mivg
6670f297d5e1408bcf417f1e
Matt Jordan
admin_assigned
2024-06-18T08:34:12.949Z
6555561fb850671a59ffacb6
/avatars/8fba2a4bb0dd71028834d5496090a4d8.svg
Matt Jordan
user
mattjordan
6670f297d5e1408bcf417f1f
Samir Gadre
admin_assigned
2024-06-18T08:34:20.755Z
62be1cd77c0388414f012f23
https://cdn-avatars.huggingface.co/v1/production/uploads/1656626356583-noauth.jpeg
Samir Gadre
user
samirg
6670f297d5e1408bcf417f20
Hritik Bansal
admin_assigned
2024-06-18T08:34:32.029Z
61c5c25705aa54027c52f7b3
/avatars/8a89e040dc331b7a83d9a704c4fc29d2.svg
Hritik Bansal
user
hbXNov
6670f297d5e1408bcf417f21
Etash Guha
admin_assigned
2024-06-18T08:34:39.530Z
65a8c9dff84e045590476618
/avatars/9caef727784cda09c7bbcd373cbe3053.svg
Etash Guha
user
EtashGuha
6670f297d5e1408bcf417f22
Sedrick Keh
admin_assigned
2024-06-18T08:34:45.392Z
6365c1158b6bd0d9f6605d3d
/avatars/1e3e6573e0cb2bb1b9d8043045120ee6.svg
Sedrick Keh
user
sedrickkeh
6670f297d5e1408bcf417f23
Kushal Arora
admin_assigned
2024-06-18T08:35:01.549Z
631b9fabf6bc4be4a65338a8
/avatars/5d52a84356cf167d2e252984bc3525aa.svg
Kushal Arora
user
kushalarora
6670f297d5e1408bcf417f24
Saurabh Garg
admin_assigned
2024-06-18T08:36:21.825Z
64a8a87e4a702899835e9b45
/avatars/b6c60f63337a78a1bf3943885f4d7a18.svg
Saurabh Garg
user
saurabhgarg
6670f297d5e1408bcf417f25
Rui Xin
6670f297d5e1408bcf417f26
Niklas Muennighoff
claimed_verified
2024-06-19T20:21:17.819Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
6670f297d5e1408bcf417f27
Reinhard Heckel
admin_assigned
2024-06-18T08:36:49.598Z
64385800c04b3b996ea3eb3b
/avatars/0bb710434cdb9822f861917a42c93414.svg
Reinhard Heckel
user
reinhardh
6670f297d5e1408bcf417f28
Jean Mercat
admin_assigned
2024-06-18T08:36:56.580Z
62be399f01dc22b4d22fc990
/avatars/3e1f3ed30773cbb410b8a9d3c854cac7.svg
Jean Mercat
user
jmercat
6670f297d5e1408bcf417f29
Mayee Chen
admin_assigned
2024-06-18T08:37:02.326Z
63096c5e5a5c889aaedf4226
/avatars/3fcc9d2b88a389471b4d5e3425c8af96.svg
Mayee Chen
user
mayee107
6670f297d5e1408bcf417f2a
Suchin Gururangan
admin_assigned
2024-06-18T08:37:08.341Z
5e9272254957053f60648a01
https://cdn-avatars.huggingface.co/v1/production/uploads/1586657211685-5e9272254957053f60648a01.png
Suchin Gururangan
user
suching
6670f297d5e1408bcf417f2b
Mitchell Wortsman
admin_assigned
2024-06-18T08:37:34.579Z
638e42f96251c8bd7ab84089
/avatars/cb7a7f902846bd9710e2c48a50ce3527.svg
Mitchell Wortsman
user
mitchellw
6670f297d5e1408bcf417f2c
Alon Albalak
admin_assigned
2024-06-18T08:37:42.592Z
611a7ec4289467cafea62d13
https://cdn-avatars.huggingface.co/v1/production/uploads/611a7ec4289467cafea62d13/pck-0fmPQkoU7yzh6-WoL.jpeg
Alon Albalak
user
alon-albalak
6670f297d5e1408bcf417f2d
Yonatan Bitton
admin_assigned
2024-06-18T08:37:55.960Z
632e0771ae0a7b1fc95630bf
https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
Yonatan
user
yonatanbitton
6670f297d5e1408bcf417f2e
Marianna Nezhurina
claimed_verified
2025-06-07T05:51:19.469Z
62500f361684e0335e527bc6
/avatars/8d2bc4c9cfa8ea82049196431cc3ebea.svg
Marianna Nezhurina
user
marianna13
6670f297d5e1408bcf417f2f
Amro Abbas
6670f297d5e1408bcf417f30
Cheng-Yu Hsieh
claimed_verified
2024-06-25T20:04:10.250Z
6408e8663461c51cf73480e4
/avatars/7e4cbdb4b933181c673c9dc9ad041767.svg
Cheng-Yu Hsieh
user
cydhsieh01
6670f297d5e1408bcf417f31
Dhruba Ghosh
6670f297d5e1408bcf417f32
Josh Gardner
claimed_verified
2024-06-21T07:07:07.299Z
63e6d6b22d2c508de90153a1
/avatars/11d653e5376bd122ff82778a8393f541.svg
Josh Gardner
user
jpgard
6670f297d5e1408bcf417f33
Maciej Kilian
6670f297d5e1408bcf417f34
Hanlin Zhang
claimed_verified
2024-12-11T09:23:32.706Z
624054bcc2c17da6a63eb539
/avatars/bf52dc0683b4100733f8696a97696d0e.svg
hlzhang109
user
hlzhang109
6670f297d5e1408bcf417f35
Rulin Shao
6670f297d5e1408bcf417f36
Sarah Pratt
6670f297d5e1408bcf417f37
Sunny Sanyal
6670f297d5e1408bcf417f38
Gabriel Ilharco
6670f297d5e1408bcf417f39
Giannis Daras
6670f297d5e1408bcf417f3a
Kalyani Marathe
6670f297d5e1408bcf417f3b
Aaron Gokaslan
6670f297d5e1408bcf417f3c
Jieyu Zhang
claimed_verified
2024-12-23T11:12:29.212Z
640131b08ba76abe4b71b5d0
/avatars/2288b96a9a0ae8f584768f54e098def1.svg
Jieyu Zhang
user
jieyuz2
6670f297d5e1408bcf417f3d
Khyathi Chandu
6670f297d5e1408bcf417f3e
Thao Nguyen
claimed_verified
2024-08-09T07:48:43.175Z
64ba4923921dbcbc96006fb8
/avatars/4eab3991e8502f72ddf8ca4825c2ad65.svg
Thao Nguyen
user
thaottn
6670f297d5e1408bcf417f3f
Igor Vasiljevic
claimed_verified
2024-06-26T08:05:17.150Z
633b2d317af633cbcd03a53e
https://cdn-avatars.huggingface.co/v1/production/uploads/633b2d317af633cbcd03a53e/awtbCqUE4qBusxuu17iFg.png
Igor Vasiljevic
user
ivas-tri
6670f297d5e1408bcf417f40
Sham Kakade
6670f297d5e1408bcf417f41
Shuran Song
6670f297d5e1408bcf417f42
Sujay Sanghavi
6670f297d5e1408bcf417f43
Fartash Faghri
claimed_verified
2024-06-18T21:27:20.579Z
65f8945a3ba58880f9794a11
/avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
Fartash Faghri
user
fartashf
6670f297d5e1408bcf417f44
Sewoong Oh
6670f297d5e1408bcf417f45
Luke Zettlemoyer
6670f297d5e1408bcf417f46
Kyle Lo
6670f297d5e1408bcf417f47
Alaaeldin El-Nouby
6670f297d5e1408bcf417f48
Hadi Pouransari
claimed_verified
2024-06-18T07:17:56.774Z
653a8e65c75dc136bfb5b0f8
/avatars/ef252794236ce0fe2debf12773c95bb2.svg
Hadi Pouransari
user
hpouransari
6670f297d5e1408bcf417f49
Alexander Toshev
6670f297d5e1408bcf417f4a
Stephanie Wang
6670f297d5e1408bcf417f4b
Dirk Groeneveld
6670f297d5e1408bcf417f4c
Luca Soldani
6670f297d5e1408bcf417f4d
Pang Wei Koh
6670f297d5e1408bcf417f4e
Jenia Jitsev
claimed_verified
2024-10-16T09:45:14.448Z
6355b485b8b79340d4630dd5
https://cdn-avatars.huggingface.co/v1/production/uploads/6355b485b8b79340d4630dd5/HIZO4ybweRy48VdCtk2MB.jpeg
Jenia Jitsev
user
JJitsev
6670f297d5e1408bcf417f4f
Thomas Kollar
6670f297d5e1408bcf417f50
Alexandros G. Dimakis
6670f297d5e1408bcf417f51
Yair Carmon
6670f297d5e1408bcf417f52
Achal Dave
6670f297d5e1408bcf417f53
Ludwig Schmidt
6670f297d5e1408bcf417f54
Vaishaal Shankar
DataComp-LM: In search of the next generation of training sets for
  language models
We introduce DataComp for Language Models (DCLM), a testbed for controlled
dataset experiments with the goal of improving language models. As part of
DCLM, we provide a standardized corpus of 240T tokens extracted from Common
Crawl, effective pretraining recipes based on the OpenLM framework, and a broad
suite of 53 downstream evaluations. Participants in the DCLM benchmark can
experiment with data curation strategies such as deduplication, filtering, and
data mixing at model scales ranging from 412M to 7B parameters. As a baseline
for DCLM, we conduct extensive experiments and find that model-based filtering
is key to assembling a high-quality training set. The resulting dataset,
DCLM-Baseline enables training a 7B parameter language model from scratch to
64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the
previous state-of-the-art in open-data language models, DCLM-Baseline
represents a 6.6 percentage point improvement on MMLU while being trained with
40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and
Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53
natural language understanding tasks while being trained with 6.6x less compute
than Llama 3 8B. Our results highlight the importance of dataset design for
training language models and offer a starting point for further research on
data curation.
DataComp for Language Models (DCLM) provides a benchmark with a standardized corpus and recipes to improve language models through controlled dataset experiments, showing the importance of data curation for performance and compute efficiency.
DataComp for Language Models DCLM standardized corpus pretraining recipes OpenLM framework downstream evaluations data curation strategies deduplication filtering data mixing model-based filtering DCLM-Baseline 5-shot accuracy MMLU MAP-Neo Mistral-7B-v0.3 Llama 3 8B natural language understanding tasks