title: Towards Best Practices for Open Datasets for LLM Training
summary: Many AI companies are training their large language models (LLMs) on data
  without the permission of the copyright owners. The permissibility of doing so
  varies by jurisdiction: in countries like the EU and Japan, this is allowed
  under certain restrictions, while in the United States, the legal landscape is
  more ambiguous. Regardless of the legal status, concerns from creative
  producers have led to several high-profile copyright lawsuits, and the threat
  of litigation is commonly cited as a reason for the recent trend towards
  minimizing the information shared about training datasets by both corporate and
  public interest actors. This trend in limiting data information causes harm by
  hindering transparency, accountability, and innovation in the broader ecosystem
  by denying researchers, auditors, and impacted individuals access to the
  information needed to understand AI models.
    While this could be mitigated by training language models on open access and
  public domain data, at the time of writing, there are no such models (trained
  at a meaningful scale) due to the substantial technical and sociological
  challenges in assembling the necessary corpus. These challenges include
  incomplete and unreliable metadata, the cost and complexity of digitizing
  physical records, and the diverse set of legal and technical skills required to
  ensure relevance and responsibility in a quickly changing landscape. Building
  towards a future where AI systems can be trained on openly licensed data that
  is responsibly curated and governed requires collaboration across legal,
  technical, and policy domains, along with investments in metadata standards,
  digitization, and fostering a culture of openness.
paper_authors._id: 6788d4566cc82aa3a079f632
paper_authors.name: Stefan Baack
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:43:49.638Z
paper_authors._id: 645954bafbf75ae1c71fb8aa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645954bafbf75ae1c71fb8aa/twyFXx2-M8SruwLwRBV1W.jpeg
paper_authors.fullname: Stefan Baack
paper_authors.type: user
paper_authors.user: stefan-baack
paper_authors._id: 6788d4566cc82aa3a079f633
paper_authors.name: Stella Biderman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:43:56.119Z
paper_authors._id: 60347d3660e3dd96631c9093
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
paper_authors.fullname: Stella Biderman
paper_authors.type: user
paper_authors.user: stellaathena
paper_authors._id: 6788d4566cc82aa3a079f634
paper_authors.name: Kasia Odrozek
paper_authors._id: 6788d4566cc82aa3a079f635
paper_authors.name: Aviya Skowron
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:44:07.420Z
paper_authors._id: 63c5dfc8d5a5cd2043e6f03c
paper_authors.avatarUrl: /avatars/edcfcd9cfb03286d670e6c5743efef6a.svg
paper_authors.fullname: Aviya Skowron
paper_authors.type: user
paper_authors.user: avi-skowron
paper_authors._id: 6788d4566cc82aa3a079f636
paper_authors.name: Ayah Bdeir
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:44:13.401Z
paper_authors._id: 66fe985ff3ba4a0ed6d2bc89
paper_authors.avatarUrl: /avatars/7ded4065561a6bd571fa94a27f328c18.svg
paper_authors.fullname: ayah bdeir
paper_authors.type: user
paper_authors.user: ayahbdeir
paper_authors._id: 6788d4566cc82aa3a079f637
paper_authors.name: Jillian Bommarito
paper_authors._id: 6788d4566cc82aa3a079f638
paper_authors.name: Jennifer Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:44:22.871Z
paper_authors._id: 62a0da842e30aaf94ebaaa12
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679073278459-62a0da842e30aaf94ebaaa12.jpeg
paper_authors.fullname: Jennifer Ding
paper_authors.type: user
paper_authors.user: jending12
paper_authors._id: 6788d4566cc82aa3a079f639
paper_authors.name: Maximilian Gahntz
paper_authors._id: 6788d4566cc82aa3a079f63a
paper_authors.name: Paul Keller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:44:50.535Z
paper_authors._id: 63692f631e9d04886d555da6
paper_authors.avatarUrl: /avatars/13035d88679a570c20c74b7325d89542.svg
paper_authors.fullname: Paul Keller
paper_authors.type: user
paper_authors.user: paulkeller
paper_authors._id: 6788d4566cc82aa3a079f63b
paper_authors.name: Pierre-Carl Langlais
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:44:59.405Z
paper_authors._id: 64ce091a9e9ca8123d7a42b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ce091a9e9ca8123d7a42b0/OEPggp82RwigxNLL35LgT.jpeg
paper_authors.fullname: Pierre-Carl Langlais
paper_authors.type: user
paper_authors.user: Pclanglais
paper_authors._id: 6788d4566cc82aa3a079f63c
paper_authors.name: Greg Lindahl
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:45:10.564Z
paper_authors._id: 656fbeae7734a829bbd16252
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656fbeae7734a829bbd16252/s0_NEIevmFM3dncq0gHPn.jpeg
paper_authors.fullname: Greg Lindahl
paper_authors.type: user
paper_authors.user: greglindahl
paper_authors._id: 6788d4566cc82aa3a079f63d
paper_authors.name: Sebastian Majstorovic
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:45:17.228Z
paper_authors._id: 636071759ddc44e710e0f5ce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636071759ddc44e710e0f5ce/-gmEhY5PidmSXIQPi2-QB.jpeg
paper_authors.fullname: Sebastian Majstorovic
paper_authors.type: user
paper_authors.user: storytracer
paper_authors._id: 6788d4566cc82aa3a079f63e
paper_authors.name: Nik Marda
paper_authors._id: 6788d4566cc82aa3a079f63f
paper_authors.name: Guilherme Penedo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:45:26.371Z
paper_authors._id: 62596f9e1c0a084224b93e00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62596f9e1c0a084224b93e00/X2aLkJ0ofhkXwAg7lXvxD.jpeg
paper_authors.fullname: Guilherme Penedo
paper_authors.type: user
paper_authors.user: guipenedo
paper_authors._id: 6788d4566cc82aa3a079f640
paper_authors.name: Maarten Van Segbroeck
paper_authors._id: 6788d4566cc82aa3a079f641
paper_authors.name: Jennifer Wang
paper_authors._id: 6788d4566cc82aa3a079f642
paper_authors.name: Leandro von Werra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:46:10.239Z
paper_authors._id: 5e48005437cb5b49818287a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
paper_authors.fullname: Leandro von Werra
paper_authors.type: user
paper_authors.user: lvwerra
paper_authors._id: 6788d4566cc82aa3a079f643
paper_authors.name: Mitchell Baker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:46:17.572Z
paper_authors._id: 63741e742b908db633716c80
paper_authors.avatarUrl: /avatars/97fa9158afab29053e47ce3067714bee.svg
paper_authors.fullname: Mitchell Baker
paper_authors.type: user
paper_authors.user: HOOisDead
paper_authors._id: 6788d4566cc82aa3a079f644
paper_authors.name: Julie Belião
paper_authors._id: 6788d4566cc82aa3a079f645
paper_authors.name: Kasia Chmielinski
paper_authors._id: 6788d4566cc82aa3a079f646
paper_authors.name: Marzieh Fadaee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:46:34.586Z
paper_authors._id: 6441042d5d600fb0951a5f99
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
paper_authors.fullname: Marzieh Fadaee
paper_authors.type: user
paper_authors.user: MarziehFadaee
paper_authors._id: 6788d4566cc82aa3a079f647
paper_authors.name: Lisa Gutermuth
paper_authors._id: 6788d4566cc82aa3a079f648
paper_authors.name: Hynek Kydlíček
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:46:45.941Z
paper_authors._id: 626ede24d2fa9e7d598c8709
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626ede24d2fa9e7d598c8709/JKS8-Y2Jw87EgNQZBRswq.jpeg
paper_authors.fullname: Hynek Kydlicek
paper_authors.type: user
paper_authors.user: hynky
paper_authors._id: 6788d4566cc82aa3a079f649
paper_authors.name: Greg Leppert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:46:51.942Z
paper_authors._id: 623b6a04ae0ec315881b9c97
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623b6a04ae0ec315881b9c97/IprOmck5cUmwKB6yoAU4L.jpeg
paper_authors.fullname: Greg Leppert
paper_authors.type: user
paper_authors.user: leppert
paper_authors._id: 6788d4566cc82aa3a079f64a
paper_authors.name: EM Lewis-Jong
paper_authors._id: 6788d4566cc82aa3a079f64b
paper_authors.name: Solana Larsen
paper_authors._id: 6788d4566cc82aa3a079f64c
paper_authors.name: Shayne Longpre
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:47:15.078Z
paper_authors._id: 61f4283a81c4d30f58140242
paper_authors.avatarUrl: /avatars/a1cf1ef1fd442c36ed65c68e51919fed.svg
paper_authors.fullname: Shayne Longpre
paper_authors.type: user
paper_authors.user: Shayne
paper_authors._id: 6788d4566cc82aa3a079f64d
paper_authors.name: Angela Oduor Lungati
paper_authors._id: 6788d4566cc82aa3a079f64e
paper_authors.name: Cullen Miller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:47:27.114Z
paper_authors._id: 6571bd30e82edf86f269fac0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6571bd30e82edf86f269fac0/310j0bc6evOUY6dmtiBq5.jpeg
paper_authors.fullname: Cullen Miller
paper_authors.type: user
paper_authors.user: cullenmiller
paper_authors._id: 6788d4566cc82aa3a079f64f
paper_authors.name: Victor Miller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:47:46.377Z
paper_authors._id: 638a93f1ed88cf97afd53e42
paper_authors.avatarUrl: /avatars/147ed42f13847b5e4d534511ef5388a3.svg
paper_authors.fullname: Victor Miller
paper_authors.type: user
paper_authors.user: victormiller
paper_authors._id: 6788d4566cc82aa3a079f650
paper_authors.name: Max Ryabinin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:47:53.155Z
paper_authors._id: 607d59fb921db717010c7ccc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625736058289-607d59fb921db717010c7ccc.png
paper_authors.fullname: Max Ryabinin
paper_authors.type: user
paper_authors.user: mryab
paper_authors._id: 6788d4566cc82aa3a079f651
paper_authors.name: Kathleen Siminyu
paper_authors._id: 6788d4566cc82aa3a079f652
paper_authors.name: Andrew Strait
paper_authors._id: 6788d4566cc82aa3a079f653
paper_authors.name: Mark Surman
paper_authors._id: 6788d4566cc82aa3a079f654
paper_authors.name: Anna Tumadóttir
paper_authors._id: 6788d4566cc82aa3a079f655
paper_authors.name: Maurice Weber
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:48:17.851Z
paper_authors._id: 6329ee3dab49d487dd1439ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6329ee3dab49d487dd1439ec/vxGvdBK0XMZaCpc5dGOIa.jpeg
paper_authors.fullname: Maurice Weber
paper_authors.type: user
paper_authors.user: mauriceweber
paper_authors._id: 6788d4566cc82aa3a079f656
paper_authors.name: Rebecca Weiss
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:48:24.368Z
paper_authors._id: 6430688e0d7e3248d0616a64
paper_authors.avatarUrl: /avatars/6d3ff97af3dd0da6f4781523e8cb2778.svg
paper_authors.fullname: Rebecca Weiss
paper_authors.type: user
paper_authors.user: rjweiss
paper_authors._id: 6788d4566cc82aa3a079f657
paper_authors.name: Lee White
paper_authors._id: 6788d4566cc82aa3a079f658
paper_authors.name: Thomas Wolf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T09:43:36.026Z
paper_authors._id: 5df7e9e5da6d0311fd3d53f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
paper_authors.fullname: Thomas Wolf
paper_authors.type: user
paper_authors.user: thomwolf
paper_title: Towards Best Practices for Open Datasets for LLM Training
paper_summary: Many AI companies are training their large language models (LLMs) on data
  without the permission of the copyright owners. The permissibility of doing so
  varies by jurisdiction: in countries like the EU and Japan, this is allowed
  under certain restrictions, while in the United States, the legal landscape is
  more ambiguous. Regardless of the legal status, concerns from creative
  producers have led to several high-profile copyright lawsuits, and the threat
  of litigation is commonly cited as a reason for the recent trend towards
  minimizing the information shared about training datasets by both corporate and
  public interest actors. This trend in limiting data information causes harm by
  hindering transparency, accountability, and innovation in the broader ecosystem
  by denying researchers, auditors, and impacted individuals access to the
  information needed to understand AI models.
    While this could be mitigated by training language models on open access and
  public domain data, at the time of writing, there are no such models (trained
  at a meaningful scale) due to the substantial technical and sociological
  challenges in assembling the necessary corpus. These challenges include
  incomplete and unreliable metadata, the cost and complexity of digitizing
  physical records, and the diverse set of legal and technical skills required to
  ensure relevance and responsibility in a quickly changing landscape. Building
  towards a future where AI systems can be trained on openly licensed data that
  is responsibly curated and governed requires collaboration across legal,
  technical, and policy domains, along with investments in metadata standards,
  digitization, and fostering a culture of openness.