title: Dolma: an Open Corpus of Three Trillion Tokens for Language Model
    Pretraining Research
summary: Language models have become a critical technology to tackling a wide range of
  natural language processing tasks, yet many details about how the
  best-performing language models were developed are not reported. In particular,
  information about their pretraining corpora is seldom discussed: commercial
  language models rarely provide any information about their data; even open
  models rarely release datasets they are trained on, or an exact recipe to
  reproduce them. As a result, it is challenging to conduct certain threads of
  language modeling research, such as understanding how training data impacts
  model capabilities and shapes their limitations. To facilitate open research on
  language model pretraining, we release Dolma, a three trillion tokens English
  corpus, built from a diverse mixture of web content, scientific papers, code,
  public-domain books, social media, and encyclopedic materials. In addition, we
  open source our data curation toolkit to enable further experimentation and
  reproduction of our work. In this report, we document Dolma, including its
  design principles, details about its construction, and a summary of its
  contents. We interleave this report with analyses and experimental results from
  training language models on intermediate states of Dolma to share what we have
  learned about important data curation practices, including the role of content
  or quality filters, deduplication, and multi-source mixing. Dolma has been used
  to train OLMo, a state-of-the-art, open language model and framework designed
  to build and study the science of language modeling.
paper_authors._id: 65bc5db5c6dea60b6b735e50
paper_authors.name: Luca Soldaini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:18:33.388Z
paper_authors._id: 5f04d8c45d08220171a0ad32
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
paper_authors.fullname: Luca Soldaini
paper_authors.type: user
paper_authors.user: soldni
paper_authors._id: 65bc5db5c6dea60b6b735e51
paper_authors.name: Rodney Kinney
paper_authors._id: 65bc5db5c6dea60b6b735e52
paper_authors.name: Akshita Bhagia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:18:48.021Z
paper_authors._id: 6282937a9624ab5f75c977e1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
paper_authors.fullname: Akshita Bhagia
paper_authors.type: user
paper_authors.user: akshitab
paper_authors._id: 65bc5db5c6dea60b6b735e53
paper_authors.name: Dustin Schwenk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:18:55.888Z
paper_authors._id: 647e791b7f9ad5e44babb965
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647e791b7f9ad5e44babb965/Vjfy4c3ooDwU24N7mvRdy.jpeg
paper_authors.fullname: Dustin Schwenk
paper_authors.type: user
paper_authors.user: drschwenk
paper_authors._id: 65bc5db5c6dea60b6b735e54
paper_authors.name: David Atkinson
paper_authors._id: 65bc5db5c6dea60b6b735e55
paper_authors.name: Russell Authur
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:19:09.218Z
paper_authors._id: 62712c97bcef985363d9e368
paper_authors.avatarUrl: /avatars/0fcc2f096bc6ba50a43cf1345d5c0066.svg
paper_authors.fullname: Russell Authur
paper_authors.type: user
paper_authors.user: rauthur
paper_authors._id: 65bc5db5c6dea60b6b735e56
paper_authors.name: Ben Bogin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:19:15.785Z
paper_authors._id: 6155cf81c0c4bdf57bbb25b4
paper_authors.avatarUrl: /avatars/84d8aa1c37972fbdcbc698e402ba799d.svg
paper_authors.fullname: Ben Bogin
paper_authors.type: user
paper_authors.user: benbogin
paper_authors._id: 65bc5db5c6dea60b6b735e57
paper_authors.name: Khyathi Chandu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:19:48.551Z
paper_authors._id: 62ebe6472f8999993a847bb6
paper_authors.avatarUrl: /avatars/29fb590f8116bdc9c55f7a333e2c4280.svg
paper_authors.fullname: Khyathi Raghavi Chandu
paper_authors.type: user
paper_authors.user: khyathi
paper_authors._id: 65bc5db5c6dea60b6b735e58
paper_authors.name: Jennifer Dumas
paper_authors._id: 65bc5db5c6dea60b6b735e59
paper_authors.name: Yanai Elazar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:20:11.377Z
paper_authors._id: 623ca115a795593324c4353f
paper_authors.avatarUrl: /avatars/bf11fe728df2786d52ed4d2de12b48d3.svg
paper_authors.fullname: Yanai Elazar
paper_authors.type: user
paper_authors.user: yanaiela
paper_authors._id: 65bc5db5c6dea60b6b735e5a
paper_authors.name: Valentin Hofmann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:20:18.621Z
paper_authors._id: 659553cd2a36333c0bf23f75
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/659553cd2a36333c0bf23f75/ZO_XLRPYKB8UgUEkZ7UwC.jpeg
paper_authors.fullname: Valentin Hofmann
paper_authors.type: user
paper_authors.user: valentinhofmann
paper_authors._id: 65bc5db5c6dea60b6b735e5b
paper_authors.name: Ananya Harsh Jha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:20:25.189Z
paper_authors._id: 6582bee321aa786b76beba44
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6582bee321aa786b76beba44/1YqJBHmLxMNRaGtU-lbKh.jpeg
paper_authors.fullname: Ananya Harsh Jha
paper_authors.type: user
paper_authors.user: ananyahjha93
paper_authors._id: 65bc5db5c6dea60b6b735e5c
paper_authors.name: Sachin Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:25:13.428Z
paper_authors._id: 64f2520f35653e77f02f288c
paper_authors.avatarUrl: /avatars/033140dce61cd5bbf849f69b8335410d.svg
paper_authors.fullname: Sachin Kumar
paper_authors.type: user
paper_authors.user: shocheen
paper_authors._id: 65bc5db5c6dea60b6b735e5d
paper_authors.name: Li Lucy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:54:06.980Z
paper_authors._id: 629a4fc60fa8aec7f1be437a
paper_authors.avatarUrl: /avatars/46045b57e6095110f4476b20776b32d8.svg
paper_authors.fullname: Lucy Li
paper_authors.type: user
paper_authors.user: lucy3
paper_authors._id: 65bc5db5c6dea60b6b735e5e
paper_authors.name: Xinxi Lyu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:21:24.000Z
paper_authors._id: 64dff1ddb5cc372803af964d
paper_authors.avatarUrl: /avatars/ed47c408897a7674a2308055e1b97fe6.svg
paper_authors.fullname: Xinxi Lyu
paper_authors.type: user
paper_authors.user: alrope
paper_authors._id: 65bc5db5c6dea60b6b735e5f
paper_authors.name: Nathan Lambert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:21:30.651Z
paper_authors._id: 628e5f90a9a3c754c1f7c88f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
paper_authors.fullname: Nathan Lambert
paper_authors.type: user
paper_authors.user: natolambert
paper_authors._id: 65bc5db5c6dea60b6b735e60
paper_authors.name: Ian Magnusson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:21:37.701Z
paper_authors._id: 62bddd0b1e22ec8427a0f27e
paper_authors.avatarUrl: /avatars/4528b10d6b94bdbee2fe3c2d28331945.svg
paper_authors.fullname: Ian Magnusson
paper_authors.type: user
paper_authors.user: IanMagnusson
paper_authors._id: 65bc5db5c6dea60b6b735e61
paper_authors.name: Jacob Morrison
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:21:53.390Z
paper_authors._id: 63f51f6cb51da4d61da72937
paper_authors.avatarUrl: /avatars/6023ecaafb28b7599f59816381d58bd0.svg
paper_authors.fullname: Jacob Morrison
paper_authors.type: user
paper_authors.user: jacobmorrison
paper_authors._id: 65bc5db5c6dea60b6b735e62
paper_authors.name: Niklas Muennighoff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:22:24.696Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_authors._id: 65bc5db5c6dea60b6b735e63
paper_authors.name: Aakanksha Naik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:22:39.700Z
paper_authors._id: 6268981926476157a9262a73
paper_authors.avatarUrl: /avatars/a6e4d81580039e8680c194d89a716535.svg
paper_authors.fullname: Aakanksha Naik
paper_authors.type: user
paper_authors.user: aakanksha
paper_authors._id: 65bc5db5c6dea60b6b735e64
paper_authors.name: Crystal Nam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:22:45.768Z
paper_authors._id: 64a7264f44387a55046f92e5
paper_authors.avatarUrl: /avatars/7f6e076a2a535957bc9c7ac4aeae716d.svg
paper_authors.fullname: Crystal Nam
paper_authors.type: user
paper_authors.user: CrystalN
paper_authors._id: 65bc5db5c6dea60b6b735e65
paper_authors.name: Matthew E. Peters
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:23:05.626Z
paper_authors._id: 63d0455a8cf6c8e4d2087eeb
paper_authors.avatarUrl: /avatars/e86cb0ffbfa3c64f162945c40a2fdb89.svg
paper_authors.fullname: Matthew Peters
paper_authors.type: user
paper_authors.user: matt-peters
paper_authors._id: 65bc5db5c6dea60b6b735e66
paper_authors.name: Abhilasha Ravichander
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:23:13.380Z
paper_authors._id: 6349886c429608888c42319a
paper_authors.avatarUrl: /avatars/f84b5fe8b76172878274754e3399d6ec.svg
paper_authors.fullname: Abhilasha Ravichander
paper_authors.type: user
paper_authors.user: lasha-nlp
paper_authors._id: 65bc5db5c6dea60b6b735e67
paper_authors.name: Kyle Richardson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-02T09:00:44.605Z
paper_authors._id: 659c6f5e1d398a238152227d
paper_authors.avatarUrl: /avatars/dba97fe8cb825102f1eae97104a71f64.svg
paper_authors.fullname: Kyle Richardson
paper_authors.type: user
paper_authors.user: yakazimir
paper_authors._id: 65bc5db5c6dea60b6b735e68
paper_authors.name: Zejiang Shen
paper_authors._id: 65bc5db5c6dea60b6b735e69
paper_authors.name: Emma Strubell
paper_authors._id: 65bc5db5c6dea60b6b735e6a
paper_authors.name: Nishant Subramani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:23:40.770Z
paper_authors._id: 617b167b941993035fbaf2bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/617b167b941993035fbaf2bf/Wt3QMbjX9EMxdIrQdrF2l.jpeg
paper_authors.fullname: Nishant Subramani
paper_authors.type: user
paper_authors.user: nsubramani23
paper_authors._id: 65bc5db5c6dea60b6b735e6b
paper_authors.name: Oyvind Tafjord
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:23:51.096Z
paper_authors._id: 609b1aeff7c1303cfd1760f3
paper_authors.avatarUrl: /avatars/3cb448514885656466886d7c84e12af1.svg
paper_authors.fullname: Oyvind Tafjord
paper_authors.type: user
paper_authors.user: OyvindTafjord
paper_authors._id: 65bc5db5c6dea60b6b735e6c
paper_authors.name: Pete Walsh
paper_authors._id: 65bc5db5c6dea60b6b735e6d
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 65bc5db5c6dea60b6b735e6e
paper_authors.name: Noah A. Smith
paper_authors._id: 65bc5db5c6dea60b6b735e6f
paper_authors.name: Hannaneh Hajishirzi
paper_authors._id: 65bc5db5c6dea60b6b735e70
paper_authors.name: Iz Beltagy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:24:17.327Z
paper_authors._id: 5ec07694ed25d76864d553f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1589816744885-5ec07694ed25d76864d553f4.jpeg
paper_authors.fullname: Iz Beltagy
paper_authors.type: user
paper_authors.user: beltagy
paper_authors._id: 65bc5db5c6dea60b6b735e71
paper_authors.name: Dirk Groeneveld
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:24:25.575Z
paper_authors._id: 60369745413a78f892e7339c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
paper_authors.fullname: Dirk Groeneveld
paper_authors.type: user
paper_authors.user: dirkgr
paper_authors._id: 65bc5db5c6dea60b6b735e72
paper_authors.name: Jesse Dodge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-02T11:24:32.424Z
paper_authors._id: 6283f38567d336d3e5d5280e
paper_authors.avatarUrl: /avatars/d0a54aaec74a90b050e671c191b87a80.svg
paper_authors.fullname: Jesse Dodge
paper_authors.type: user
paper_authors.user: JesseDodge
paper_authors._id: 65bc5db5c6dea60b6b735e73
paper_authors.name: Kyle Lo
paper_title: Dolma: an Open Corpus of Three Trillion Tokens for Language Model
    Pretraining Research
paper_summary: Language models have become a critical technology to tackling a wide range of
  natural language processing tasks, yet many details about how the
  best-performing language models were developed are not reported. In particular,
  information about their pretraining corpora is seldom discussed: commercial
  language models rarely provide any information about their data; even open
  models rarely release datasets they are trained on, or an exact recipe to
  reproduce them. As a result, it is challenging to conduct certain threads of
  language modeling research, such as understanding how training data impacts
  model capabilities and shapes their limitations. To facilitate open research on
  language model pretraining, we release Dolma, a three trillion tokens English
  corpus, built from a diverse mixture of web content, scientific papers, code,
  public-domain books, social media, and encyclopedic materials. In addition, we
  open source our data curation toolkit to enable further experimentation and
  reproduction of our work. In this report, we document Dolma, including its
  design principles, details about its construction, and a summary of its
  contents. We interleave this report with analyses and experimental results from
  training language models on intermediate states of Dolma to share what we have
  learned about important data curation practices, including the role of content
  or quality filters, deduplication, and multi-source mixing. Dolma has been used
  to train OLMo, a state-of-the-art, open language model and framework designed
  to build and study the science of language modeling.
paper_ai_keywords: natural language processing language models pretraining corpora data curation data curation toolkit web content scientific papers open-domain books social media encyclopedic materials content filters deduplication multi-source mixing OLMo open language model language modeling framework science of language modeling