title: The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly
    Licensed Text
summary: Large language models (LLMs) are typically trained on enormous quantities of
  unlicensed text, a practice that has led to scrutiny due to possible
  intellectual property infringement and ethical concerns. Training LLMs on
  openly licensed text presents a first step towards addressing these issues, but
  prior data collection efforts have yielded datasets too small or low-quality to
  produce performant LLMs. To address this gap, we collect, curate, and release
  the Common Pile v0.1, an eight terabyte collection of openly licensed text
  designed for LLM pretraining. The Common Pile comprises content from 30 sources
  that span diverse domains including research papers, code, books,
  encyclopedias, educational materials, audio transcripts, and more. Crucially,
  we validate our efforts by training two 7 billion parameter LLMs on text from
  the Common Pile: Comma v0.1-1T and Comma v0.1-2T, trained on 1 and 2 trillion
  tokens respectively. Both models attain competitive performance to LLMs trained
  on unlicensed text with similar computational budgets, such as Llama 1 and 2
  7B. In addition to releasing the Common Pile v0.1 itself, we also release the
  code used in its creation as well as the training mixture and checkpoints for
  the Comma v0.1 models.
paper_authors._id: 684247f35d537e0e5ecb724b
paper_authors.name: Nikhil Kandpal
paper_authors._id: 684247f35d537e0e5ecb724c
paper_authors.name: Brian Lester
paper_authors._id: 684247f35d537e0e5ecb724d
paper_authors.name: Colin Raffel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:49:07.948Z
paper_authors._id: 6079c29765b9d0165cb18392
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
paper_authors.fullname: Colin Raffel
paper_authors.type: user
paper_authors.user: craffel
paper_authors._id: 684247f35d537e0e5ecb724e
paper_authors.name: Sebastian Majstorovic
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:41:37.270Z
paper_authors._id: 636071759ddc44e710e0f5ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636071759ddc44e710e0f5ce/-gmEhY5PidmSXIQPi2-QB.jpeg
paper_authors.fullname: Sebastian Majstorovic
paper_authors.type: user
paper_authors.user: storytracer
paper_authors._id: 684247f35d537e0e5ecb724f
paper_authors.name: Stella Biderman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:48:59.510Z
paper_authors._id: 60347d3660e3dd96631c9093
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
paper_authors.fullname: Stella Biderman
paper_authors.type: user
paper_authors.user: stellaathena
paper_authors._id: 684247f35d537e0e5ecb7250
paper_authors.name: Baber Abbasi
paper_authors._id: 684247f35d537e0e5ecb7251
paper_authors.name: Luca Soldaini
paper_authors._id: 684247f35d537e0e5ecb7252
paper_authors.name: Enrico Shippole
paper_authors._id: 684247f35d537e0e5ecb7253
paper_authors.name: A. Feder Cooper
paper_authors._id: 684247f35d537e0e5ecb7254
paper_authors.name: Aviya Skowron
paper_authors._id: 684247f35d537e0e5ecb7255
paper_authors.name: John Kirchenbauer
paper_authors._id: 684247f35d537e0e5ecb7256
paper_authors.name: Shayne Longpre
paper_authors._id: 684247f35d537e0e5ecb7257
paper_authors.name: Lintang Sutawika
paper_authors._id: 684247f35d537e0e5ecb7258
paper_authors.name: Alon Albalak
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:49:05.890Z
paper_authors._id: 611a7ec4289467cafea62d13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/611a7ec4289467cafea62d13/pck-0fmPQkoU7yzh6-WoL.jpeg
paper_authors.fullname: Alon Albalak
paper_authors.type: user
paper_authors.user: alon-albalak
paper_authors._id: 684247f35d537e0e5ecb7259
paper_authors.name: Zhenlin Xu
paper_authors._id: 684247f35d537e0e5ecb725a
paper_authors.name: Guilherme Penedo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:49:10.010Z
paper_authors._id: 62596f9e1c0a084224b93e00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62596f9e1c0a084224b93e00/X2aLkJ0ofhkXwAg7lXvxD.jpeg
paper_authors.fullname: Guilherme Penedo
paper_authors.type: user
paper_authors.user: guipenedo
paper_authors._id: 684247f35d537e0e5ecb725b
paper_authors.name: Loubna Ben Allal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T13:33:58.334Z
paper_authors._id: 61c141342aac764ce1654e43
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61c141342aac764ce1654e43/81AwoT5IQ_Xdw0OVw7TKu.jpeg
paper_authors.fullname: Loubna Ben Allal
paper_authors.type: user
paper_authors.user: loubnabnl
paper_authors._id: 684247f35d537e0e5ecb725c
paper_authors.name: Elie Bakouch
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:49:01.927Z
paper_authors._id: 651e96991b97c9f33d26bde6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651e96991b97c9f33d26bde6/-Bqs6qrmz0yCfwtB2e-6q.jpeg
paper_authors.fullname: Elie Bakouch
paper_authors.type: user
paper_authors.user: eliebak
paper_authors._id: 684247f35d537e0e5ecb725d
paper_authors.name: John David Pressman
paper_authors._id: 684247f35d537e0e5ecb725e
paper_authors.name: Honglu Fan
paper_authors._id: 684247f35d537e0e5ecb725f
paper_authors.name: Dashiell Stander
paper_authors._id: 684247f35d537e0e5ecb7260
paper_authors.name: Guangyu Song
paper_authors._id: 684247f35d537e0e5ecb7261
paper_authors.name: Aaron Gokaslan
paper_authors._id: 684247f35d537e0e5ecb7262
paper_authors.name: Tom Goldstein
paper_authors._id: 684247f35d537e0e5ecb7263
paper_authors.name: Brian R. Bartoldson
paper_authors._id: 684247f35d537e0e5ecb7264
paper_authors.name: Bhavya Kailkhura
paper_authors._id: 684247f35d537e0e5ecb7265
paper_authors.name: Tyler Murray
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:49:03.957Z
paper_authors._id: 65e5fefbe8cbae176d9ca005
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65e5fefbe8cbae176d9ca005/5SLREDsVzycEwVsPNv765.jpeg
paper_authors.fullname: Tyler Murray
paper_authors.type: user
paper_authors.user: undfined
paper_title: The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly
    Licensed Text
paper_summary: Large language models (LLMs) are typically trained on enormous quantities of
  unlicensed text, a practice that has led to scrutiny due to possible
  intellectual property infringement and ethical concerns. Training LLMs on
  openly licensed text presents a first step towards addressing these issues, but
  prior data collection efforts have yielded datasets too small or low-quality to
  produce performant LLMs. To address this gap, we collect, curate, and release
  the Common Pile v0.1, an eight terabyte collection of openly licensed text
  designed for LLM pretraining. The Common Pile comprises content from 30 sources
  that span diverse domains including research papers, code, books,
  encyclopedias, educational materials, audio transcripts, and more. Crucially,
  we validate our efforts by training two 7 billion parameter LLMs on text from
  the Common Pile: Comma v0.1-1T and Comma v0.1-2T, trained on 1 and 2 trillion
  tokens respectively. Both models attain competitive performance to LLMs trained
  on unlicensed text with similar computational budgets, such as Llama 1 and 2
  7B. In addition to releasing the Common Pile v0.1 itself, we also release the
  code used in its creation as well as the training mixture and checkpoints for
  the Comma v0.1 models.
paper_ai_summary: The Common Pile v0.1 dataset, an 8-terabyte collection of openly licensed text, is used to train competitive 7 billion parameter LLMs.
paper_ai_keywords: Large language models LLMs openly licensed text Common Pile v0.1 parameter-efficient fine-tuning Llama 1 and 2 7B training mixture checkpoints