title: RedPajama: an Open Dataset for Training Large Language Models
summary: Large language models are increasingly becoming a cornerstone technology in
  artificial intelligence, the sciences, and society as a whole, yet the optimal
  strategies for dataset composition and filtering remain largely elusive. Many
  of the top-performing models lack transparency in their dataset curation and
  model development processes, posing an obstacle to the development of fully
  open language models. In this paper, we identify three core data-related
  challenges that must be addressed to advance open-source language models. These
  include (1) transparency in model development, including the data curation
  process, (2) access to large quantities of high-quality data, and (3)
  availability of artifacts and metadata for dataset curation and analysis. To
  address these challenges, we release RedPajama-V1, an open reproduction of the
  LLaMA training dataset. In addition, we release RedPajama-V2, a massive
  web-only dataset consisting of raw, unfiltered text data together with quality
  signals and metadata. Together, the RedPajama datasets comprise over 100
  trillion tokens spanning multiple domains and with their quality signals
  facilitate the filtering of data, aiming to inspire the development of numerous
  new datasets. To date, these datasets have already been used in the training of
  strong language models used in production, such as Snowflake Arctic,
  Salesforce's XGen and AI2's OLMo. To provide insight into the quality of
  RedPajama, we present a series of analyses and ablation studies with
  decoder-only language models with up to 1.6B parameters. Our findings
  demonstrate how quality signals for web data can be effectively leveraged to
  curate high-quality subsets of the dataset, underscoring the potential of
  RedPajama to advance the development of transparent and high-performing
  language models at scale.
paper_authors._id: 673db8720916efb6cf79a413
paper_authors.name: Maurice Weber
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-20T13:20:10.244Z
paper_authors._id: 6329ee3dab49d487dd1439ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6329ee3dab49d487dd1439ec/vxGvdBK0XMZaCpc5dGOIa.jpeg
paper_authors.fullname: Maurice Weber
paper_authors.type: user
paper_authors.user: mauriceweber
paper_authors._id: 673db8720916efb6cf79a414
paper_authors.name: Daniel Fu
paper_authors._id: 673db8720916efb6cf79a415
paper_authors.name: Quentin Anthony
paper_authors._id: 673db8720916efb6cf79a416
paper_authors.name: Yonatan Oren
paper_authors._id: 673db8720916efb6cf79a417
paper_authors.name: Shane Adams
paper_authors._id: 673db8720916efb6cf79a418
paper_authors.name: Anton Alexandrov
paper_authors._id: 673db8720916efb6cf79a419
paper_authors.name: Xiaozhong Lyu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-20T13:30:20.737Z
paper_authors._id: 6512adc7d4556f9e6508e89f
paper_authors.avatarUrl: /avatars/fbae9a9092d6c02ef49e60bc38c6640c.svg
paper_authors.fullname: xiaozhong lyu
paper_authors.type: user
paper_authors.user: xiaozhong-idata
paper_authors._id: 673db8720916efb6cf79a41a
paper_authors.name: Huu Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:19:30.683Z
paper_authors._id: 5fc6879e1c5ee87b1164876d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fc6879e1c5ee87b1164876d/Tjnm_lv0Bq0gPbFOTDH6E.jpeg
paper_authors.fullname: Huu Nguyen
paper_authors.type: user
paper_authors.user: huu-ontocord
paper_authors._id: 673db8720916efb6cf79a41b
paper_authors.name: Xiaozhe Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-20T13:28:26.931Z
paper_authors._id: 61919e59d221f4281b3833d5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669164541734-61919e59d221f4281b3833d5.jpeg
paper_authors.fullname: Xiaozhe Yao
paper_authors.type: user
paper_authors.user: xzyao
paper_authors._id: 673db8720916efb6cf79a41c
paper_authors.name: Virginia Adams
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-20T13:28:43.998Z
paper_authors._id: 664d56756083f276c4197f46
paper_authors.avatarUrl: /avatars/64032612a471a0d53f5d04d89617f393.svg
paper_authors.fullname: Adams
paper_authors.type: user
paper_authors.user: VirginiaAdams
paper_authors._id: 673db8720916efb6cf79a41d
paper_authors.name: Ben Athiwaratkun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-20T13:29:45.831Z
paper_authors._id: 6395124b3d608ea4d9cd2339
paper_authors.avatarUrl: /avatars/13f3dc014495dfdef2c460e9ce9c9869.svg
paper_authors.fullname: Ben Athiwaratkun
paper_authors.type: user
paper_authors.user: benathi
paper_authors._id: 673db8720916efb6cf79a41e
paper_authors.name: Rahul Chalamala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-20T13:30:03.317Z
paper_authors._id: 648abf5f0c2a2d8592ca07e7
paper_authors.avatarUrl: /avatars/333a96e0b2e9420b2abacf538c776c18.svg
paper_authors.fullname: Rahul Chalamala
paper_authors.type: user
paper_authors.user: rchalamala
paper_authors._id: 673db8720916efb6cf79a41f
paper_authors.name: Kezhen Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-20T13:30:10.106Z
paper_authors._id: 65caa69b8870f8c093a8abc1
paper_authors.avatarUrl: /avatars/1b76dcc6a31b827a5b597024a0f29164.svg
paper_authors.fullname: Kezhen Chen
paper_authors.type: user
paper_authors.user: kezhentogether
paper_authors._id: 673db8720916efb6cf79a420
paper_authors.name: Max Ryabinin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-20T13:27:41.096Z
paper_authors._id: 607d59fb921db717010c7ccc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625736058289-607d59fb921db717010c7ccc.png
paper_authors.fullname: Max Ryabinin
paper_authors.type: user
paper_authors.user: mryab
paper_authors._id: 673db8720916efb6cf79a421
paper_authors.name: Tri Dao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-20T13:27:47.803Z
paper_authors._id: 64b8a6b5cf14c2fabe98159b
paper_authors.avatarUrl: /avatars/dbc009451865435bf290791beadc4723.svg
paper_authors.fullname: Tri Dao
paper_authors.type: user
paper_authors.user: tridao
paper_authors._id: 673db8720916efb6cf79a422
paper_authors.name: Percy Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-20T13:27:53.332Z
paper_authors._id: 6409651b9e9f790c905b2335
paper_authors.avatarUrl: /avatars/1fb8c80b60f21f65a0a027319101f236.svg
paper_authors.fullname: Percy Liang
paper_authors.type: user
paper_authors.user: percyliang
paper_authors._id: 673db8720916efb6cf79a423
paper_authors.name: Christopher Ré
paper_authors._id: 673db8720916efb6cf79a424
paper_authors.name: Irina Rish
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-20T13:23:17.710Z
paper_authors._id: 6227f764236b7b2eefa7f173
paper_authors.avatarUrl: /avatars/0204e3da8e9ef12d88c98c6e955f8340.svg
paper_authors.fullname: Irina Rish
paper_authors.type: user
paper_authors.user: irinarish
paper_authors._id: 673db8720916efb6cf79a425
paper_authors.name: Ce Zhang
paper_title: RedPajama: an Open Dataset for Training Large Language Models
paper_summary: Large language models are increasingly becoming a cornerstone technology in
  artificial intelligence, the sciences, and society as a whole, yet the optimal
  strategies for dataset composition and filtering remain largely elusive. Many
  of the top-performing models lack transparency in their dataset curation and
  model development processes, posing an obstacle to the development of fully
  open language models. In this paper, we identify three core data-related
  challenges that must be addressed to advance open-source language models. These
  include (1) transparency in model development, including the data curation
  process, (2) access to large quantities of high-quality data, and (3)
  availability of artifacts and metadata for dataset curation and analysis. To
  address these challenges, we release RedPajama-V1, an open reproduction of the
  LLaMA training dataset. In addition, we release RedPajama-V2, a massive
  web-only dataset consisting of raw, unfiltered text data together with quality
  signals and metadata. Together, the RedPajama datasets comprise over 100
  trillion tokens spanning multiple domains and with their quality signals
  facilitate the filtering of data, aiming to inspire the development of numerous
  new datasets. To date, these datasets have already been used in the training of
  strong language models used in production, such as Snowflake Arctic,
  Salesforce's XGen and AI2's OLMo. To provide insight into the quality of
  RedPajama, we present a series of analyses and ablation studies with
  decoder-only language models with up to 1.6B parameters. Our findings
  demonstrate how quality signals for web data can be effectively leveraged to
  curate high-quality subsets of the dataset, underscoring the potential of
  RedPajama to advance the development of transparent and high-performing
  language models at scale.
paper_ai_summary: The RedPajama datasets are introduced to address core challenges for open-source language models by providing transparent data curation, large volumes of high-quality text, and quality signals for web data analysis.
paper_ai_keywords: decoder-only language models