Towards Best Practices for Open Datasets for LLM Training
Many AI companies are training their large language models (LLMs) on data
without the permission of the copyright owners. The permissibility of doing so
varies by jurisdiction: in countries like the EU and Japan, this is allowed
under certain restrictions, while in the United States, the legal landscape is
more ambiguous. Regardless of the legal status, concerns from creative
producers have led to several high-profile copyright lawsuits, and the threat
of litigation is commonly cited as a reason for the recent trend towards
minimizing the information shared about training datasets by both corporate and
public interest actors. This trend in limiting data information causes harm by
hindering transparency, accountability, and innovation in the broader ecosystem
by denying researchers, auditors, and impacted individuals access to the
information needed to understand AI models.
  While this could be mitigated by training language models on open access and
public domain data, at the time of writing, there are no such models (trained
at a meaningful scale) due to the substantial technical and sociological
challenges in assembling the necessary corpus. These challenges include
incomplete and unreliable metadata, the cost and complexity of digitizing
physical records, and the diverse set of legal and technical skills required to
ensure relevance and responsibility in a quickly changing landscape. Building
towards a future where AI systems can be trained on openly licensed data that
is responsibly curated and governed requires collaboration across legal,
technical, and policy domains, along with investments in metadata standards,
digitization, and fostering a culture of openness.
6788d4566cc82aa3a079f632
Stefan Baack
admin_assigned
2025-01-16T09:43:49.638Z
645954bafbf75ae1c71fb8aa
https://cdn-avatars.huggingface.co/v1/production/uploads/645954bafbf75ae1c71fb8aa/W1fzXtRhm1NC_VMHyZoCJ.jpeg
Stefan Baack
user
stefan-baack
6788d4566cc82aa3a079f633
Stella Biderman
admin_assigned
2025-01-16T09:43:56.119Z
60347d3660e3dd96631c9093
https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
Stella Biderman
user
stellaathena
6788d4566cc82aa3a079f634
Kasia Odrozek
6788d4566cc82aa3a079f635
Aviya Skowron
admin_assigned
2025-01-16T09:44:07.420Z
63c5dfc8d5a5cd2043e6f03c
/avatars/edcfcd9cfb03286d670e6c5743efef6a.svg
Aviya Skowron
user
avi-skowron
6788d4566cc82aa3a079f636
Ayah Bdeir
admin_assigned
2025-01-16T09:44:13.401Z
66fe985ff3ba4a0ed6d2bc89
/avatars/7ded4065561a6bd571fa94a27f328c18.svg
ayah bdeir
user
ayahbdeir
6788d4566cc82aa3a079f637
Jillian Bommarito
6788d4566cc82aa3a079f638
Jennifer Ding
admin_assigned
2025-01-16T09:44:22.871Z
62a0da842e30aaf94ebaaa12
https://cdn-avatars.huggingface.co/v1/production/uploads/1679073278459-62a0da842e30aaf94ebaaa12.jpeg
Jennifer Ding
user
jending12
6788d4566cc82aa3a079f639
Maximilian Gahntz
claimed_verified
2025-07-31T08:17:02.334Z
688a320df3af1c9c13d375b7
https://cdn-avatars.huggingface.co/v1/production/uploads/688a320df3af1c9c13d375b7/txgDd7fz9PVoVzA7Wu5HP.png
Maximilian Gahntz
user
mxghntz
6788d4566cc82aa3a079f63a
Paul Keller
admin_assigned
2025-01-16T09:44:50.535Z
63692f631e9d04886d555da6
/avatars/13035d88679a570c20c74b7325d89542.svg
Paul Keller
user
paulkeller
6788d4566cc82aa3a079f63b
Pierre-Carl Langlais
admin_assigned
2025-01-16T09:44:59.405Z
64ce091a9e9ca8123d7a42b0
https://cdn-avatars.huggingface.co/v1/production/uploads/64ce091a9e9ca8123d7a42b0/OEPggp82RwigxNLL35LgT.jpeg
Pierre-Carl Langlais
user
Pclanglais
6788d4566cc82aa3a079f63c
Greg Lindahl
admin_assigned
2025-01-16T09:45:10.564Z
656fbeae7734a829bbd16252
https://cdn-avatars.huggingface.co/v1/production/uploads/656fbeae7734a829bbd16252/s0_NEIevmFM3dncq0gHPn.jpeg
Greg Lindahl
user
greglindahl
6788d4566cc82aa3a079f63d
Sebastian Majstorovic
admin_assigned
2025-01-16T09:45:17.228Z
636071759ddc44e710e0f5ce
https://cdn-avatars.huggingface.co/v1/production/uploads/636071759ddc44e710e0f5ce/-gmEhY5PidmSXIQPi2-QB.jpeg
Sebastian Majstorovic
user
storytracer
6788d4566cc82aa3a079f63e
Nik Marda
6788d4566cc82aa3a079f63f
Guilherme Penedo
admin_assigned
2025-01-16T09:45:26.371Z
62596f9e1c0a084224b93e00
https://cdn-avatars.huggingface.co/v1/production/uploads/62596f9e1c0a084224b93e00/X2aLkJ0ofhkXwAg7lXvxD.jpeg
Guilherme Penedo
user
guipenedo
6788d4566cc82aa3a079f640
Maarten Van Segbroeck
6788d4566cc82aa3a079f641
Jennifer Wang
6788d4566cc82aa3a079f642
Leandro von Werra
admin_assigned
2025-01-16T09:46:10.239Z
5e48005437cb5b49818287a5
https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
Leandro von Werra
user
lvwerra
6788d4566cc82aa3a079f643
Mitchell Baker
admin_assigned
2025-01-16T09:46:17.572Z
63741e742b908db633716c80
/avatars/97fa9158afab29053e47ce3067714bee.svg
Mitchell Baker
user
HOOisDead
6788d4566cc82aa3a079f644
Julie Belião
6788d4566cc82aa3a079f645
Kasia Chmielinski
6788d4566cc82aa3a079f646
Marzieh Fadaee
admin_assigned
2025-01-16T09:46:34.586Z
6441042d5d600fb0951a5f99
https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
Marzieh Fadaee
user
MarziehFadaee
6788d4566cc82aa3a079f647
Lisa Gutermuth
6788d4566cc82aa3a079f648
Hynek Kydlíček
admin_assigned
2025-01-16T09:46:45.941Z
626ede24d2fa9e7d598c8709
https://cdn-avatars.huggingface.co/v1/production/uploads/626ede24d2fa9e7d598c8709/JKS8-Y2Jw87EgNQZBRswq.jpeg
Hynek Kydlicek
user
hynky
6788d4566cc82aa3a079f649
Greg Leppert
admin_assigned
2025-01-16T09:46:51.942Z
623b6a04ae0ec315881b9c97
https://cdn-avatars.huggingface.co/v1/production/uploads/623b6a04ae0ec315881b9c97/IprOmck5cUmwKB6yoAU4L.jpeg
Greg Leppert
user
leppert
6788d4566cc82aa3a079f64a
EM Lewis-Jong
6788d4566cc82aa3a079f64b
Solana Larsen
6788d4566cc82aa3a079f64c
Shayne Longpre
admin_assigned
2025-01-16T09:47:15.078Z
61f4283a81c4d30f58140242
/avatars/a1cf1ef1fd442c36ed65c68e51919fed.svg
Shayne Longpre
user
Shayne
6788d4566cc82aa3a079f64d
Angela Oduor Lungati
6788d4566cc82aa3a079f64e
Cullen Miller
admin_assigned
2025-01-16T09:47:27.114Z
6571bd30e82edf86f269fac0
https://cdn-avatars.huggingface.co/v1/production/uploads/6571bd30e82edf86f269fac0/310j0bc6evOUY6dmtiBq5.jpeg
Cullen Miller
user
cullenmiller
6788d4566cc82aa3a079f64f
Victor Miller
admin_assigned
2025-01-16T09:47:46.377Z
638a93f1ed88cf97afd53e42
/avatars/147ed42f13847b5e4d534511ef5388a3.svg
Victor Miller
user
victormiller
6788d4566cc82aa3a079f650
Max Ryabinin
admin_assigned
2025-01-16T09:47:53.155Z
607d59fb921db717010c7ccc
https://cdn-avatars.huggingface.co/v1/production/uploads/1625736058289-607d59fb921db717010c7ccc.png
Max Ryabinin
user
mryab
6788d4566cc82aa3a079f651
Kathleen Siminyu
6788d4566cc82aa3a079f652
Andrew Strait
6788d4566cc82aa3a079f653
Mark Surman
6788d4566cc82aa3a079f654
Anna Tumadóttir
6788d4566cc82aa3a079f655
Maurice Weber
admin_assigned
2025-01-16T09:48:17.851Z
6329ee3dab49d487dd1439ec
https://cdn-avatars.huggingface.co/v1/production/uploads/6329ee3dab49d487dd1439ec/vxGvdBK0XMZaCpc5dGOIa.jpeg
Maurice Weber
user
mauriceweber
6788d4566cc82aa3a079f656
Rebecca Weiss
admin_assigned
2025-01-16T09:48:24.368Z
6430688e0d7e3248d0616a64
/avatars/6d3ff97af3dd0da6f4781523e8cb2778.svg
Rebecca Weiss
user
rjweiss
6788d4566cc82aa3a079f657
Lee White
6788d4566cc82aa3a079f658
Thomas Wolf
admin_assigned
2025-01-16T09:43:36.026Z
5df7e9e5da6d0311fd3d53f9
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
Thomas Wolf
user
thomwolf
Towards Best Practices for Open Datasets for LLM Training
Many AI companies are training their large language models (LLMs) on data
without the permission of the copyright owners. The permissibility of doing so
varies by jurisdiction: in countries like the EU and Japan, this is allowed
under certain restrictions, while in the United States, the legal landscape is
more ambiguous. Regardless of the legal status, concerns from creative
producers have led to several high-profile copyright lawsuits, and the threat
of litigation is commonly cited as a reason for the recent trend towards
minimizing the information shared about training datasets by both corporate and
public interest actors. This trend in limiting data information causes harm by
hindering transparency, accountability, and innovation in the broader ecosystem
by denying researchers, auditors, and impacted individuals access to the
information needed to understand AI models.
  While this could be mitigated by training language models on open access and
public domain data, at the time of writing, there are no such models (trained
at a meaningful scale) due to the substantial technical and sociological
challenges in assembling the necessary corpus. These challenges include
incomplete and unreliable metadata, the cost and complexity of digitizing
physical records, and the diverse set of legal and technical skills required to
ensure relevance and responsibility in a quickly changing landscape. Building
towards a future where AI systems can be trained on openly licensed data that
is responsibly curated and governed requires collaboration across legal,
technical, and policy domains, along with investments in metadata standards,
digitization, and fostering a culture of openness.