Dolma: an Open Corpus of Three Trillion Tokens for Language Model
  Pretraining Research
Language models have become a critical technology to tackling a wide range of
natural language processing tasks, yet many details about how the
best-performing language models were developed are not reported. In particular,
information about their pretraining corpora is seldom discussed: commercial
language models rarely provide any information about their data; even open
models rarely release datasets they are trained on, or an exact recipe to
reproduce them. As a result, it is challenging to conduct certain threads of
language modeling research, such as understanding how training data impacts
model capabilities and shapes their limitations. To facilitate open research on
language model pretraining, we release Dolma, a three trillion tokens English
corpus, built from a diverse mixture of web content, scientific papers, code,
public-domain books, social media, and encyclopedic materials. In addition, we
open source our data curation toolkit to enable further experimentation and
reproduction of our work. In this report, we document Dolma, including its
design principles, details about its construction, and a summary of its
contents. We interleave this report with analyses and experimental results from
training language models on intermediate states of Dolma to share what we have
learned about important data curation practices, including the role of content
or quality filters, deduplication, and multi-source mixing. Dolma has been used
to train OLMo, a state-of-the-art, open language model and framework designed
to build and study the science of language modeling.
65bc5db5c6dea60b6b735e50
Luca Soldaini
admin_assigned
2024-02-02T11:18:33.388Z
5f04d8c45d08220171a0ad32
https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
Luca Soldaini
user
soldni
65bc5db5c6dea60b6b735e51
Rodney Kinney
65bc5db5c6dea60b6b735e52
Akshita Bhagia
admin_assigned
2024-02-02T11:18:48.021Z
6282937a9624ab5f75c977e1
https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
Akshita Bhagia
user
akshitab
65bc5db5c6dea60b6b735e53
Dustin Schwenk
admin_assigned
2024-02-02T11:18:55.888Z
647e791b7f9ad5e44babb965
https://cdn-avatars.huggingface.co/v1/production/uploads/647e791b7f9ad5e44babb965/Vjfy4c3ooDwU24N7mvRdy.jpeg
Dustin Schwenk
user
drschwenk
65bc5db5c6dea60b6b735e54
David Atkinson
65bc5db5c6dea60b6b735e55
Russell Authur
admin_assigned
2024-02-02T11:19:09.218Z
62712c97bcef985363d9e368
/avatars/0fcc2f096bc6ba50a43cf1345d5c0066.svg
Russell Authur
user
rauthur
65bc5db5c6dea60b6b735e56
Ben Bogin
admin_assigned
2024-02-02T11:19:15.785Z
6155cf81c0c4bdf57bbb25b4
/avatars/84d8aa1c37972fbdcbc698e402ba799d.svg
Ben Bogin
user
benbogin
65bc5db5c6dea60b6b735e57
Khyathi Chandu
admin_assigned
2024-02-02T11:19:48.551Z
62ebe6472f8999993a847bb6
/avatars/29fb590f8116bdc9c55f7a333e2c4280.svg
Khyathi Raghavi Chandu
user
khyathi
65bc5db5c6dea60b6b735e58
Jennifer Dumas
65bc5db5c6dea60b6b735e59
Yanai Elazar
admin_assigned
2024-02-02T11:20:11.377Z
623ca115a795593324c4353f
/avatars/bf11fe728df2786d52ed4d2de12b48d3.svg
Yanai Elazar
user
yanaiela
65bc5db5c6dea60b6b735e5a
Valentin Hofmann
admin_assigned
2024-02-02T11:20:18.621Z
659553cd2a36333c0bf23f75
https://cdn-avatars.huggingface.co/v1/production/uploads/659553cd2a36333c0bf23f75/7ojOQ_26rRg3cn0_qGNSC.jpeg
Valentin Hofmann
user
valentinhofmann
65bc5db5c6dea60b6b735e5b
Ananya Harsh Jha
admin_assigned
2024-02-02T11:20:25.189Z
6582bee321aa786b76beba44
https://cdn-avatars.huggingface.co/v1/production/uploads/6582bee321aa786b76beba44/1YqJBHmLxMNRaGtU-lbKh.jpeg
Ananya Harsh Jha
user
ananyahjha93
65bc5db5c6dea60b6b735e5c
Sachin Kumar
claimed_verified
2024-07-19T11:25:13.428Z
64f2520f35653e77f02f288c
/avatars/33620ae12f604e63eb9ec5ee9d89a820.svg
Sachin Kumar
user
shocheen
65bc5db5c6dea60b6b735e5d
Li Lucy
claimed_verified
2024-11-15T09:54:06.980Z
629a4fc60fa8aec7f1be437a
/avatars/46045b57e6095110f4476b20776b32d8.svg
Lucy Li
user
lucy3
65bc5db5c6dea60b6b735e5e
Xinxi Lyu
admin_assigned
2024-02-02T11:21:24.000Z
64dff1ddb5cc372803af964d
/avatars/ed47c408897a7674a2308055e1b97fe6.svg
Xinxi Lyu
user
alrope
65bc5db5c6dea60b6b735e5f
Nathan Lambert
admin_assigned
2024-02-02T11:21:30.651Z
628e5f90a9a3c754c1f7c88f
https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
Nathan Lambert
user
natolambert
65bc5db5c6dea60b6b735e60
Ian Magnusson
admin_assigned
2024-02-02T11:21:37.701Z
62bddd0b1e22ec8427a0f27e
/avatars/c433d0fb2945c70d1998f1e0e98d265f.svg
Ian Magnusson
user
IanMagnusson
65bc5db5c6dea60b6b735e61
Jacob Morrison
admin_assigned
2024-02-02T11:21:53.390Z
63f51f6cb51da4d61da72937
/avatars/6023ecaafb28b7599f59816381d58bd0.svg
Jacob Morrison
user
jacobmorrison
65bc5db5c6dea60b6b735e62
Niklas Muennighoff
admin_assigned
2024-02-02T11:22:24.696Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
65bc5db5c6dea60b6b735e63
Aakanksha Naik
admin_assigned
2024-02-02T11:22:39.700Z
6268981926476157a9262a73
/avatars/a6e4d81580039e8680c194d89a716535.svg
Aakanksha Naik
user
aakanksha
65bc5db5c6dea60b6b735e64
Crystal Nam
admin_assigned
2024-02-02T11:22:45.768Z
64a7264f44387a55046f92e5
/avatars/7f6e076a2a535957bc9c7ac4aeae716d.svg
Crystal Nam
user
CrystalN
65bc5db5c6dea60b6b735e65
Matthew E. Peters
admin_assigned
2024-02-02T11:23:05.626Z
63d0455a8cf6c8e4d2087eeb
/avatars/e86cb0ffbfa3c64f162945c40a2fdb89.svg
Matthew Peters
user
matt-peters
65bc5db5c6dea60b6b735e66
Abhilasha Ravichander
admin_assigned
2024-02-02T11:23:13.380Z
6349886c429608888c42319a
/avatars/f84b5fe8b76172878274754e3399d6ec.svg
Abhilasha Ravichander
user
lasha-nlp
65bc5db5c6dea60b6b735e67
Kyle Richardson
claimed_verified
2024-02-02T09:00:44.605Z
659c6f5e1d398a238152227d
/avatars/dba97fe8cb825102f1eae97104a71f64.svg
Kyle Richardson
user
yakazimir
65bc5db5c6dea60b6b735e68
Zejiang Shen
65bc5db5c6dea60b6b735e69
Emma Strubell
65bc5db5c6dea60b6b735e6a
Nishant Subramani
admin_assigned
2024-02-02T11:23:40.770Z
617b167b941993035fbaf2bf
https://cdn-avatars.huggingface.co/v1/production/uploads/617b167b941993035fbaf2bf/Wt3QMbjX9EMxdIrQdrF2l.jpeg
Nishant Subramani
user
nsubramani23
65bc5db5c6dea60b6b735e6b
Oyvind Tafjord
admin_assigned
2024-02-02T11:23:51.096Z
609b1aeff7c1303cfd1760f3
/avatars/3cb448514885656466886d7c84e12af1.svg
Oyvind Tafjord
user
OyvindTafjord
65bc5db5c6dea60b6b735e6c
Pete Walsh
65bc5db5c6dea60b6b735e6d
Luke Zettlemoyer
65bc5db5c6dea60b6b735e6e
Noah A. Smith
65bc5db5c6dea60b6b735e6f
Hannaneh Hajishirzi
65bc5db5c6dea60b6b735e70
Iz Beltagy
admin_assigned
2024-02-02T11:24:17.327Z
5ec07694ed25d76864d553f4
https://cdn-avatars.huggingface.co/v1/production/uploads/1589816744885-5ec07694ed25d76864d553f4.jpeg
Iz Beltagy
user
beltagy
65bc5db5c6dea60b6b735e71
Dirk Groeneveld
admin_assigned
2024-02-02T11:24:25.575Z
60369745413a78f892e7339c
https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
Dirk Groeneveld
user
dirkgr
65bc5db5c6dea60b6b735e72
Jesse Dodge
admin_assigned
2024-02-02T11:24:32.424Z
6283f38567d336d3e5d5280e
/avatars/d0a54aaec74a90b050e671c191b87a80.svg
Jesse Dodge
user
JesseDodge
65bc5db5c6dea60b6b735e73
Kyle Lo
Dolma: an Open Corpus of Three Trillion Tokens for Language Model
  Pretraining Research
Language models have become a critical technology to tackling a wide range of
natural language processing tasks, yet many details about how the
best-performing language models were developed are not reported. In particular,
information about their pretraining corpora is seldom discussed: commercial
language models rarely provide any information about their data; even open
models rarely release datasets they are trained on, or an exact recipe to
reproduce them. As a result, it is challenging to conduct certain threads of
language modeling research, such as understanding how training data impacts
model capabilities and shapes their limitations. To facilitate open research on
language model pretraining, we release Dolma, a three trillion tokens English
corpus, built from a diverse mixture of web content, scientific papers, code,
public-domain books, social media, and encyclopedic materials. In addition, we
open source our data curation toolkit to enable further experimentation and
reproduction of our work. In this report, we document Dolma, including its
design principles, details about its construction, and a summary of its
contents. We interleave this report with analyses and experimental results from
training language models on intermediate states of Dolma to share what we have
learned about important data curation practices, including the role of content
or quality filters, deduplication, and multi-source mixing. Dolma has been used
to train OLMo, a state-of-the-art, open language model and framework designed
to build and study the science of language modeling.
Dolma, a three trillion token English corpus, and its open-source data curation toolkit facilitate open research in language model pretraining by providing detailed insights into data curation practices.
language models Dolma data curation English corpus web content scientific papers public-domain books social media encyclopedic materials data curation toolkit OLMo language model pretraining content filters deduplication multi-source mixing