title: What's In My Big Data?
summary: Large text corpora are the backbone of language models. However, we have a
  limited understanding of the content of these corpora, including general
  statistics, quality, social factors, and inclusion of evaluation data
  (contamination). In this work, we propose What's In My Big Data? (WIMBD), a
  platform and a set of sixteen analyses that allow us to reveal and compare the
  contents of large text corpora. WIMBD builds on two basic capabilities -- count
  and search -- at scale, which allows us to analyze more than 35 terabytes on a
  standard compute node. We apply WIMBD to ten different corpora used to train
  popular language models, including C4, The Pile, and RedPajama. Our analysis
  uncovers several surprising and previously undocumented findings about these
  corpora, including the high prevalence of duplicate, synthetic, and low-quality
  content, personally identifiable information, toxic language, and benchmark
  contamination. For instance, we find that about 50% of the documents in
  RedPajama and LAION-2B-en are duplicates. In addition, several datasets used
  for benchmarking models trained on such corpora are contaminated with respect
  to important benchmarks, including the Winograd Schema Challenge and parts of
  GLUE and SuperGLUE. We open-source WIMBD's code and artifacts to provide a
  standard set of evaluations for new text-based corpora and to encourage more
  analyses and transparency around them: github.com/allenai/wimbd.
paper_authors._id: 6541c93cea69bcb62029a415
paper_authors.name: Yanai Elazar
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-11-01T04:57:17.205Z
paper_authors._id: 623ca115a795593324c4353f
paper_authors.avatarUrl: /avatars/bf11fe728df2786d52ed4d2de12b48d3.svg
paper_authors.fullname: Yanai Elazar
paper_authors.type: user
paper_authors.user: yanaiela
paper_authors._id: 6541c93cea69bcb62029a416
paper_authors.name: Akshita Bhagia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-19T19:21:41.372Z
paper_authors._id: 6282937a9624ab5f75c977e1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
paper_authors.fullname: Akshita Bhagia
paper_authors.type: user
paper_authors.user: akshitab
paper_authors._id: 6541c93cea69bcb62029a417
paper_authors.name: Ian Magnusson
paper_authors._id: 6541c93cea69bcb62029a418
paper_authors.name: Abhilasha Ravichander
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T17:15:18.215Z
paper_authors._id: 6349886c429608888c42319a
paper_authors.avatarUrl: /avatars/f84b5fe8b76172878274754e3399d6ec.svg
paper_authors.fullname: Abhilasha Ravichander
paper_authors.type: user
paper_authors.user: lasha-nlp
paper_authors._id: 6541c93cea69bcb62029a419
paper_authors.name: Dustin Schwenk
paper_authors._id: 6541c93cea69bcb62029a41a
paper_authors.name: Alane Suhr
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-09T20:07:45.497Z
paper_authors._id: 6611e6e1188ff298b0dd0b79
paper_authors.avatarUrl: /avatars/3a495283955ec9e06e1829c7eb2cd9a4.svg
paper_authors.fullname: Alane Suhr
paper_authors.type: user
paper_authors.user: alsuhr
paper_authors._id: 6541c93cea69bcb62029a41b
paper_authors.name: Pete Walsh
paper_authors._id: 6541c93cea69bcb62029a41c
paper_authors.name: Dirk Groeneveld
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-04T09:34:18.870Z
paper_authors._id: 60369745413a78f892e7339c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
paper_authors.fullname: Dirk Groeneveld
paper_authors.type: user
paper_authors.user: dirkgr
paper_authors._id: 6541c93cea69bcb62029a41d
paper_authors.name: Luca Soldaini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-25T09:44:54.820Z
paper_authors._id: 5f04d8c45d08220171a0ad32
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
paper_authors.fullname: Luca Soldaini
paper_authors.type: user
paper_authors.user: soldni
paper_authors._id: 6541c93cea69bcb62029a41e
paper_authors.name: Sameer Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-12T19:58:39.986Z
paper_authors._id: 60b1206e6dfba500226fbf58
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1622220905791-noauth.png
paper_authors.fullname: Sameer Singh
paper_authors.type: user
paper_authors.user: sameer
paper_authors._id: 6541c93cea69bcb62029a41f
paper_authors.name: Hanna Hajishirzi
paper_authors._id: 6541c93cea69bcb62029a420
paper_authors.name: Noah A. Smith
paper_authors._id: 6541c93cea69bcb62029a421
paper_authors.name: Jesse Dodge
paper_title: What's In My Big Data?
paper_summary: Large text corpora are the backbone of language models. However, we have a
  limited understanding of the content of these corpora, including general
  statistics, quality, social factors, and inclusion of evaluation data
  (contamination). In this work, we propose What's In My Big Data? (WIMBD), a
  platform and a set of sixteen analyses that allow us to reveal and compare the
  contents of large text corpora. WIMBD builds on two basic capabilities -- count
  and search -- at scale, which allows us to analyze more than 35 terabytes on a
  standard compute node. We apply WIMBD to ten different corpora used to train
  popular language models, including C4, The Pile, and RedPajama. Our analysis
  uncovers several surprising and previously undocumented findings about these
  corpora, including the high prevalence of duplicate, synthetic, and low-quality
  content, personally identifiable information, toxic language, and benchmark
  contamination. For instance, we find that about 50% of the documents in
  RedPajama and LAION-2B-en are duplicates. In addition, several datasets used
  for benchmarking models trained on such corpora are contaminated with respect
  to important benchmarks, including the Winograd Schema Challenge and parts of
  GLUE and SuperGLUE. We open-source WIMBD's code and artifacts to provide a
  standard set of evaluations for new text-based corpora and to encourage more
  analyses and transparency around them: github.com/allenai/wimbd.
paper_ai_summary: WIMBD is a platform for analyzing large text corpora, revealing duplicates, synthetic content, and contamination in datasets used for language models.
paper_ai_keywords: WIMBD large text corpora language models text-based corpora