title: DataComp-LM: In search of the next generation of training sets for
    language models
summary: We introduce DataComp for Language Models (DCLM), a testbed for controlled
  dataset experiments with the goal of improving language models. As part of
  DCLM, we provide a standardized corpus of 240T tokens extracted from Common
  Crawl, effective pretraining recipes based on the OpenLM framework, and a broad
  suite of 53 downstream evaluations. Participants in the DCLM benchmark can
  experiment with data curation strategies such as deduplication, filtering, and
  data mixing at model scales ranging from 412M to 7B parameters. As a baseline
  for DCLM, we conduct extensive experiments and find that model-based filtering
  is key to assembling a high-quality training set. The resulting dataset,
  DCLM-Baseline enables training a 7B parameter language model from scratch to
  64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the
  previous state-of-the-art in open-data language models, DCLM-Baseline
  represents a 6.6 percentage point improvement on MMLU while being trained with
  40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and
  Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53
  natural language understanding tasks while being trained with 6.6x less compute
  than Llama 3 8B. Our results highlight the importance of dataset design for
  training language models and offer a starting point for further research on
  data curation.
paper_authors._id: 6670f297d5e1408bcf417f1a
paper_authors.name: Jeffrey Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:32:22.843Z
paper_authors._id: 63ca51a82ffd350b485f9fa5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674203513156-noauth.jpeg
paper_authors.fullname: Jeffrey Li
paper_authors.type: user
paper_authors.user: JeffreyLi
paper_authors._id: 6670f297d5e1408bcf417f1b
paper_authors.name: Alex Fang
paper_authors._id: 6670f297d5e1408bcf417f1c
paper_authors.name: Georgios Smyrnis
paper_authors._id: 6670f297d5e1408bcf417f1d
paper_authors.name: Maor Ivgi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T08:20:07.624Z
paper_authors._id: 610b7f53b1d06c607ab4436d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/610b7f53b1d06c607ab4436d/Bqw7fIsjpuukFP11S-awA.jpeg
paper_authors.fullname: Maor
paper_authors.type: user
paper_authors.user: Mivg
paper_authors._id: 6670f297d5e1408bcf417f1e
paper_authors.name: Matt Jordan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:34:12.949Z
paper_authors._id: 6555561fb850671a59ffacb6
paper_authors.avatarUrl: /avatars/8fba2a4bb0dd71028834d5496090a4d8.svg
paper_authors.fullname: Matt Jordan
paper_authors.type: user
paper_authors.user: mattjordan
paper_authors._id: 6670f297d5e1408bcf417f1f
paper_authors.name: Samir Gadre
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:34:20.755Z
paper_authors._id: 62be1cd77c0388414f012f23
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656626356583-noauth.jpeg
paper_authors.fullname: Samir Gadre
paper_authors.type: user
paper_authors.user: samirg
paper_authors._id: 6670f297d5e1408bcf417f20
paper_authors.name: Hritik Bansal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:34:32.029Z
paper_authors._id: 61c5c25705aa54027c52f7b3
paper_authors.avatarUrl: /avatars/8a89e040dc331b7a83d9a704c4fc29d2.svg
paper_authors.fullname: Hritik Bansal
paper_authors.type: user
paper_authors.user: hbXNov
paper_authors._id: 6670f297d5e1408bcf417f21
paper_authors.name: Etash Guha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:34:39.530Z
paper_authors._id: 65a8c9dff84e045590476618
paper_authors.avatarUrl: /avatars/9caef727784cda09c7bbcd373cbe3053.svg
paper_authors.fullname: Etash Guha
paper_authors.type: user
paper_authors.user: EtashGuha
paper_authors._id: 6670f297d5e1408bcf417f22
paper_authors.name: Sedrick Keh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:34:45.392Z
paper_authors._id: 6365c1158b6bd0d9f6605d3d
paper_authors.avatarUrl: /avatars/1e3e6573e0cb2bb1b9d8043045120ee6.svg
paper_authors.fullname: Sedrick Keh
paper_authors.type: user
paper_authors.user: sedrickkeh
paper_authors._id: 6670f297d5e1408bcf417f23
paper_authors.name: Kushal Arora
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:35:01.549Z
paper_authors._id: 631b9fabf6bc4be4a65338a8
paper_authors.avatarUrl: /avatars/5d52a84356cf167d2e252984bc3525aa.svg
paper_authors.fullname: Kushal Arora
paper_authors.type: user
paper_authors.user: kushalarora
paper_authors._id: 6670f297d5e1408bcf417f24
paper_authors.name: Saurabh Garg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:36:21.825Z
paper_authors._id: 64a8a87e4a702899835e9b45
paper_authors.avatarUrl: /avatars/b6c60f63337a78a1bf3943885f4d7a18.svg
paper_authors.fullname: Saurabh Garg
paper_authors.type: user
paper_authors.user: saurabhgarg
paper_authors._id: 6670f297d5e1408bcf417f25
paper_authors.name: Rui Xin
paper_authors._id: 6670f297d5e1408bcf417f26
paper_authors.name: Niklas Muennighoff
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:21:17.819Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_authors._id: 6670f297d5e1408bcf417f27
paper_authors.name: Reinhard Heckel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:36:49.598Z
paper_authors._id: 64385800c04b3b996ea3eb3b
paper_authors.avatarUrl: /avatars/0bb710434cdb9822f861917a42c93414.svg
paper_authors.fullname: Reinhard Heckel
paper_authors.type: user
paper_authors.user: reinhardh
paper_authors._id: 6670f297d5e1408bcf417f28
paper_authors.name: Jean Mercat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:36:56.580Z
paper_authors._id: 62be399f01dc22b4d22fc990
paper_authors.avatarUrl: /avatars/3e1f3ed30773cbb410b8a9d3c854cac7.svg
paper_authors.fullname: Jean Mercat
paper_authors.type: user
paper_authors.user: jmercat
paper_authors._id: 6670f297d5e1408bcf417f29
paper_authors.name: Mayee Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:37:02.326Z
paper_authors._id: 63096c5e5a5c889aaedf4226
paper_authors.avatarUrl: /avatars/3fcc9d2b88a389471b4d5e3425c8af96.svg
paper_authors.fullname: Mayee Chen
paper_authors.type: user
paper_authors.user: mayee107
paper_authors._id: 6670f297d5e1408bcf417f2a
paper_authors.name: Suchin Gururangan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:37:08.341Z
paper_authors._id: 5e9272254957053f60648a01
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1586657211685-5e9272254957053f60648a01.png
paper_authors.fullname: Suchin Gururangan
paper_authors.type: user
paper_authors.user: suching
paper_authors._id: 6670f297d5e1408bcf417f2b
paper_authors.name: Mitchell Wortsman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:37:34.579Z
paper_authors._id: 638e42f96251c8bd7ab84089
paper_authors.avatarUrl: /avatars/cb7a7f902846bd9710e2c48a50ce3527.svg
paper_authors.fullname: Mitchell Wortsman
paper_authors.type: user
paper_authors.user: mitchellw
paper_authors._id: 6670f297d5e1408bcf417f2c
paper_authors.name: Alon Albalak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:37:42.592Z
paper_authors._id: 611a7ec4289467cafea62d13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/611a7ec4289467cafea62d13/pck-0fmPQkoU7yzh6-WoL.jpeg
paper_authors.fullname: Alon Albalak
paper_authors.type: user
paper_authors.user: alon-albalak
paper_authors._id: 6670f297d5e1408bcf417f2d
paper_authors.name: Yonatan Bitton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:37:55.960Z
paper_authors._id: 632e0771ae0a7b1fc95630bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
paper_authors.fullname: Yonatan
paper_authors.type: user
paper_authors.user: yonatanbitton
paper_authors._id: 6670f297d5e1408bcf417f2e
paper_authors.name: Marianna Nezhurina
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:51:19.469Z
paper_authors._id: 62500f361684e0335e527bc6
paper_authors.avatarUrl: /avatars/8d2bc4c9cfa8ea82049196431cc3ebea.svg
paper_authors.fullname: Marianna Nezhurina
paper_authors.type: user
paper_authors.user: marianna13
paper_authors._id: 6670f297d5e1408bcf417f2f
paper_authors.name: Amro Abbas
paper_authors._id: 6670f297d5e1408bcf417f30
paper_authors.name: Cheng-Yu Hsieh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T20:04:10.250Z
paper_authors._id: 6408e8663461c51cf73480e4
paper_authors.avatarUrl: /avatars/7e4cbdb4b933181c673c9dc9ad041767.svg
paper_authors.fullname: Cheng-Yu Hsieh
paper_authors.type: user
paper_authors.user: cydhsieh01
paper_authors._id: 6670f297d5e1408bcf417f31
paper_authors.name: Dhruba Ghosh
paper_authors._id: 6670f297d5e1408bcf417f32
paper_authors.name: Josh Gardner
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:07:07.299Z
paper_authors._id: 63e6d6b22d2c508de90153a1
paper_authors.avatarUrl: /avatars/11d653e5376bd122ff82778a8393f541.svg
paper_authors.fullname: Josh Gardner
paper_authors.type: user
paper_authors.user: jpgard
paper_authors._id: 6670f297d5e1408bcf417f33
paper_authors.name: Maciej Kilian
paper_authors._id: 6670f297d5e1408bcf417f34
paper_authors.name: Hanlin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:23:32.706Z
paper_authors._id: 624054bcc2c17da6a63eb539
paper_authors.avatarUrl: /avatars/bf52dc0683b4100733f8696a97696d0e.svg
paper_authors.fullname: hlzhang109
paper_authors.type: user
paper_authors.user: hlzhang109
paper_authors._id: 6670f297d5e1408bcf417f35
paper_authors.name: Rulin Shao
paper_authors._id: 6670f297d5e1408bcf417f36
paper_authors.name: Sarah Pratt
paper_authors._id: 6670f297d5e1408bcf417f37
paper_authors.name: Sunny Sanyal
paper_authors._id: 6670f297d5e1408bcf417f38
paper_authors.name: Gabriel Ilharco
paper_authors._id: 6670f297d5e1408bcf417f39
paper_authors.name: Giannis Daras
paper_authors._id: 6670f297d5e1408bcf417f3a
paper_authors.name: Kalyani Marathe
paper_authors._id: 6670f297d5e1408bcf417f3b
paper_authors.name: Aaron Gokaslan
paper_authors._id: 6670f297d5e1408bcf417f3c
paper_authors.name: Jieyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:12:29.212Z
paper_authors._id: 640131b08ba76abe4b71b5d0
paper_authors.avatarUrl: /avatars/2288b96a9a0ae8f584768f54e098def1.svg
paper_authors.fullname: Jieyu Zhang
paper_authors.type: user
paper_authors.user: jieyuz2
paper_authors._id: 6670f297d5e1408bcf417f3d
paper_authors.name: Khyathi Chandu
paper_authors._id: 6670f297d5e1408bcf417f3e
paper_authors.name: Thao Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:48:43.175Z
paper_authors._id: 64ba4923921dbcbc96006fb8
paper_authors.avatarUrl: /avatars/4eab3991e8502f72ddf8ca4825c2ad65.svg
paper_authors.fullname: Thao Nguyen
paper_authors.type: user
paper_authors.user: thaottn
paper_authors._id: 6670f297d5e1408bcf417f3f
paper_authors.name: Igor Vasiljevic
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:05:17.150Z
paper_authors._id: 633b2d317af633cbcd03a53e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633b2d317af633cbcd03a53e/awtbCqUE4qBusxuu17iFg.png
paper_authors.fullname: Igor Vasiljevic
paper_authors.type: user
paper_authors.user: ivas-tri
paper_authors._id: 6670f297d5e1408bcf417f40
paper_authors.name: Sham Kakade
paper_authors._id: 6670f297d5e1408bcf417f41
paper_authors.name: Shuran Song
paper_authors._id: 6670f297d5e1408bcf417f42
paper_authors.name: Sujay Sanghavi
paper_authors._id: 6670f297d5e1408bcf417f43
paper_authors.name: Fartash Faghri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T21:27:20.579Z
paper_authors._id: 65f8945a3ba58880f9794a11
paper_authors.avatarUrl: /avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
paper_authors.fullname: Fartash Faghri
paper_authors.type: user
paper_authors.user: fartashf
paper_authors._id: 6670f297d5e1408bcf417f44
paper_authors.name: Sewoong Oh
paper_authors._id: 6670f297d5e1408bcf417f45
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 6670f297d5e1408bcf417f46
paper_authors.name: Kyle Lo
paper_authors._id: 6670f297d5e1408bcf417f47
paper_authors.name: Alaaeldin El-Nouby
paper_authors._id: 6670f297d5e1408bcf417f48
paper_authors.name: Hadi Pouransari
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:17:56.774Z
paper_authors._id: 653a8e65c75dc136bfb5b0f8
paper_authors.avatarUrl: /avatars/ef252794236ce0fe2debf12773c95bb2.svg
paper_authors.fullname: Hadi Pouransari
paper_authors.type: user
paper_authors.user: hpouransari
paper_authors._id: 6670f297d5e1408bcf417f49
paper_authors.name: Alexander Toshev
paper_authors._id: 6670f297d5e1408bcf417f4a
paper_authors.name: Stephanie Wang
paper_authors._id: 6670f297d5e1408bcf417f4b
paper_authors.name: Dirk Groeneveld
paper_authors._id: 6670f297d5e1408bcf417f4c
paper_authors.name: Luca Soldani
paper_authors._id: 6670f297d5e1408bcf417f4d
paper_authors.name: Pang Wei Koh
paper_authors._id: 6670f297d5e1408bcf417f4e
paper_authors.name: Jenia Jitsev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:45:14.448Z
paper_authors._id: 6355b485b8b79340d4630dd5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6355b485b8b79340d4630dd5/HIZO4ybweRy48VdCtk2MB.jpeg
paper_authors.fullname: Jenia Jitsev
paper_authors.type: user
paper_authors.user: JJitsev
paper_authors._id: 6670f297d5e1408bcf417f4f
paper_authors.name: Thomas Kollar
paper_authors._id: 6670f297d5e1408bcf417f50
paper_authors.name: Alexandros G. Dimakis
paper_authors._id: 6670f297d5e1408bcf417f51
paper_authors.name: Yair Carmon
paper_authors._id: 6670f297d5e1408bcf417f52
paper_authors.name: Achal Dave
paper_authors._id: 6670f297d5e1408bcf417f53
paper_authors.name: Ludwig Schmidt
paper_authors._id: 6670f297d5e1408bcf417f54
paper_authors.name: Vaishaal Shankar
paper_title: DataComp-LM: In search of the next generation of training sets for
    language models
paper_summary: We introduce DataComp for Language Models (DCLM), a testbed for controlled
  dataset experiments with the goal of improving language models. As part of
  DCLM, we provide a standardized corpus of 240T tokens extracted from Common
  Crawl, effective pretraining recipes based on the OpenLM framework, and a broad
  suite of 53 downstream evaluations. Participants in the DCLM benchmark can
  experiment with data curation strategies such as deduplication, filtering, and
  data mixing at model scales ranging from 412M to 7B parameters. As a baseline
  for DCLM, we conduct extensive experiments and find that model-based filtering
  is key to assembling a high-quality training set. The resulting dataset,
  DCLM-Baseline enables training a 7B parameter language model from scratch to
  64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the
  previous state-of-the-art in open-data language models, DCLM-Baseline
  represents a 6.6 percentage point improvement on MMLU while being trained with
  40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and
  Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53
  natural language understanding tasks while being trained with 6.6x less compute
  than Llama 3 8B. Our results highlight the importance of dataset design for
  training language models and offer a starting point for further research on
  data curation.
paper_ai_summary: DataComp for Language Models (DCLM) provides a benchmark with a standardized corpus and recipes to improve language models through controlled dataset experiments, showing the importance of data curation for performance and compute efficiency.
paper_ai_keywords: DataComp for Language Models DCLM standardized corpus pretraining recipes OpenLM framework downstream evaluations data curation strategies deduplication filtering data mixing model-based filtering DCLM-Baseline 5-shot accuracy MMLU MAP-Neo Mistral-7B-v0.3 Llama 3 8B natural language understanding tasks