title: Bridging the Data Provenance Gap Across Text, Speech and Video
summary: Progress in AI is driven largely by the scale and quality of training data.
  Despite this, there is a deficit of empirical analysis examining the attributes
  of well-established datasets beyond text. In this work we conduct the largest
  and first-of-its-kind longitudinal audit across modalities--popular text,
  speech, and video datasets--from their detailed sourcing trends and use
  restrictions to their geographical and linguistic representation. Our manual
  analysis covers nearly 4000 public datasets between 1990-2024, spanning 608
  languages, 798 sources, 659 organizations, and 67 countries. We find that
  multimodal machine learning applications have overwhelmingly turned to
  web-crawled, synthetic, and social media platforms, such as YouTube, for their
  training sets, eclipsing all other sources since 2019. Secondly, tracing the
  chain of dataset derivations we find that while less than 33% of datasets are
  restrictively licensed, over 80% of the source content in widely-used text,
  speech, and video datasets, carry non-commercial restrictions. Finally, counter
  to the rising number of languages and geographies represented in public AI
  training datasets, our audit demonstrates measures of relative geographical and
  multilingual representation have failed to significantly improve their coverage
  since 2013. We believe the breadth of our audit enables us to empirically
  examine trends in data sourcing, restrictions, and Western-centricity at an
  ecosystem-level, and that visibility into these questions are essential to
  progress in responsible AI. As a contribution to ongoing improvements in
  dataset transparency and responsible use, we release our entire multimodal
  audit, allowing practitioners to trace data provenance across text, speech, and
  video.
paper_authors._id: 676b6f6e1f5ca46174ac96c8
paper_authors.name: Shayne Longpre
paper_authors._id: 676b6f6e1f5ca46174ac96c9
paper_authors.name: Nikhil Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:47:22.961Z
paper_authors._id: 6374384c250cf1379bc278f8
paper_authors.avatarUrl: /avatars/e436988005ad4f9d99fcafb8bdfade9c.svg
paper_authors.fullname: Nikhil Singh
paper_authors.type: user
paper_authors.user: nsingh1
paper_authors._id: 676b6f6e1f5ca46174ac96ca
paper_authors.name: Manuel Cherep
paper_authors._id: 676b6f6e1f5ca46174ac96cb
paper_authors.name: Kushagra Tiwary
paper_authors._id: 676b6f6e1f5ca46174ac96cc
paper_authors.name: Joanna Materzynska
paper_authors._id: 676b6f6e1f5ca46174ac96cd
paper_authors.name: William Brannon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:15:29.452Z
paper_authors._id: 641bdd25a63c4e8062387b6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641bdd25a63c4e8062387b6a/dFanBhG6_NVqpRNB-jjC4.png
paper_authors.fullname: William Brannon
paper_authors.type: user
paper_authors.user: wwbrannon
paper_authors._id: 676b6f6e1f5ca46174ac96ce
paper_authors.name: Robert Mahari
paper_authors._id: 676b6f6e1f5ca46174ac96cf
paper_authors.name: Manan Dey
paper_authors._id: 676b6f6e1f5ca46174ac96d0
paper_authors.name: Mohammed Hamdy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:33:07.482Z
paper_authors._id: 62645f88c39850dc093d6105
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650745211725-noauth.png
paper_authors.fullname: Mohammed Hamdy
paper_authors.type: user
paper_authors.user: mmhamdy
paper_authors._id: 676b6f6e1f5ca46174ac96d1
paper_authors.name: Nayan Saxena
paper_authors._id: 676b6f6e1f5ca46174ac96d2
paper_authors.name: Ahmad Mustafa Anis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:33:00.926Z
paper_authors._id: 6246908d8031dcfa9ef6d80b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6246908d8031dcfa9ef6d80b/hVdURjUl1RS2MZf4qOhvI.jpeg
paper_authors.fullname: Ahmad Mustafa Anis
paper_authors.type: user
paper_authors.user: AhmadMustafa
paper_authors._id: 676b6f6e1f5ca46174ac96d3
paper_authors.name: Emad A. Alghamdi
paper_authors._id: 676b6f6e1f5ca46174ac96d4
paper_authors.name: Vu Minh Chien
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-12T09:19:03.796Z
paper_authors._id: 60535c9d10aba34e3b6a2ef7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1616075850230-noauth.jpeg
paper_authors.fullname: vumichien
paper_authors.type: user
paper_authors.user: vumichien
paper_authors._id: 676b6f6e1f5ca46174ac96d5
paper_authors.name: Naana Obeng-Marnu
paper_authors._id: 676b6f6e1f5ca46174ac96d6
paper_authors.name: Da Yin
paper_authors._id: 676b6f6e1f5ca46174ac96d7
paper_authors.name: Kun Qian
paper_authors._id: 676b6f6e1f5ca46174ac96d8
paper_authors.name: Yizhi Li
paper_authors._id: 676b6f6e1f5ca46174ac96d9
paper_authors.name: Minnie Liang
paper_authors._id: 676b6f6e1f5ca46174ac96da
paper_authors.name: An Dinh
paper_authors._id: 676b6f6e1f5ca46174ac96db
paper_authors.name: Shrestha Mohanty
paper_authors._id: 676b6f6e1f5ca46174ac96dc
paper_authors.name: Deividas Mataciunas
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:33:05.129Z
paper_authors._id: 6040a00558b78f3a0047c23a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6040a00558b78f3a0047c23a/_BsyJoaCBO3r6GnfSgnIA.jpeg
paper_authors.fullname: David Mataciunas
paper_authors.type: user
paper_authors.user: DeividasM
paper_authors._id: 676b6f6e1f5ca46174ac96dd
paper_authors.name: Tobin South
paper_authors._id: 676b6f6e1f5ca46174ac96de
paper_authors.name: Jianguo Zhang
paper_authors._id: 676b6f6e1f5ca46174ac96df
paper_authors.name: Ariel N. Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T16:54:01.989Z
paper_authors._id: 638bcfa91987d67b340e6c1c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638bcfa91987d67b340e6c1c/3tHCB_J6c4-lsEZ_zJSlp.jpeg
paper_authors.fullname: Ariel N. Lee
paper_authors.type: user
paper_authors.user: arielnlee
paper_authors._id: 676b6f6e1f5ca46174ac96e0
paper_authors.name: Campbell S. Lund
paper_authors._id: 676b6f6e1f5ca46174ac96e1
paper_authors.name: Christopher Klamm
paper_authors._id: 676b6f6e1f5ca46174ac96e2
paper_authors.name: Damien Sileo
paper_authors._id: 676b6f6e1f5ca46174ac96e3
paper_authors.name: Diganta Misra
paper_authors._id: 676b6f6e1f5ca46174ac96e4
paper_authors.name: Enrico Shippole
paper_authors._id: 676b6f6e1f5ca46174ac96e5
paper_authors.name: Kevin Klyman
paper_authors._id: 676b6f6e1f5ca46174ac96e6
paper_authors.name: Lester JV Miranda
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-06T08:00:50.985Z
paper_authors._id: 634e20a0c1ce28f1de920cc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
paper_authors.fullname: Lj V. Miranda
paper_authors.type: user
paper_authors.user: ljvmiranda921
paper_authors._id: 676b6f6e1f5ca46174ac96e7
paper_authors.name: Niklas Muennighoff
paper_authors._id: 676b6f6e1f5ca46174ac96e8
paper_authors.name: Seonghyeon Ye
paper_authors._id: 676b6f6e1f5ca46174ac96e9
paper_authors.name: Seungone Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:33:02.972Z
paper_authors._id: 6469949654873f0043b09c22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
paper_authors.fullname: Seungone Kim
paper_authors.type: user
paper_authors.user: seungone
paper_authors._id: 676b6f6e1f5ca46174ac96ea
paper_authors.name: Vipul Gupta
paper_authors._id: 676b6f6e1f5ca46174ac96eb
paper_authors.name: Vivek Sharma
paper_authors._id: 676b6f6e1f5ca46174ac96ec
paper_authors.name: Xuhui Zhou
paper_authors._id: 676b6f6e1f5ca46174ac96ed
paper_authors.name: Caiming Xiong
paper_authors._id: 676b6f6e1f5ca46174ac96ee
paper_authors.name: Luis Villa
paper_authors._id: 676b6f6e1f5ca46174ac96ef
paper_authors.name: Stella Biderman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:50:33.769Z
paper_authors._id: 60347d3660e3dd96631c9093
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
paper_authors.fullname: Stella Biderman
paper_authors.type: user
paper_authors.user: stellaathena
paper_authors._id: 676b6f6e1f5ca46174ac96f0
paper_authors.name: Alex Pentland
paper_authors._id: 676b6f6e1f5ca46174ac96f1
paper_authors.name: Sara Hooker
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-20T09:30:34.082Z
paper_authors._id: 63434eb76f59b79da07dbddf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
paper_authors.fullname: Sara Hooker
paper_authors.type: user
paper_authors.user: sarahooker
paper_authors._id: 676b6f6e1f5ca46174ac96f2
paper_authors.name: Jad Kabbara
paper_title: Bridging the Data Provenance Gap Across Text, Speech and Video
paper_summary: Progress in AI is driven largely by the scale and quality of training data.
  Despite this, there is a deficit of empirical analysis examining the attributes
  of well-established datasets beyond text. In this work we conduct the largest
  and first-of-its-kind longitudinal audit across modalities--popular text,
  speech, and video datasets--from their detailed sourcing trends and use
  restrictions to their geographical and linguistic representation. Our manual
  analysis covers nearly 4000 public datasets between 1990-2024, spanning 608
  languages, 798 sources, 659 organizations, and 67 countries. We find that
  multimodal machine learning applications have overwhelmingly turned to
  web-crawled, synthetic, and social media platforms, such as YouTube, for their
  training sets, eclipsing all other sources since 2019. Secondly, tracing the
  chain of dataset derivations we find that while less than 33% of datasets are
  restrictively licensed, over 80% of the source content in widely-used text,
  speech, and video datasets, carry non-commercial restrictions. Finally, counter
  to the rising number of languages and geographies represented in public AI
  training datasets, our audit demonstrates measures of relative geographical and
  multilingual representation have failed to significantly improve their coverage
  since 2013. We believe the breadth of our audit enables us to empirically
  examine trends in data sourcing, restrictions, and Western-centricity at an
  ecosystem-level, and that visibility into these questions are essential to
  progress in responsible AI. As a contribution to ongoing improvements in
  dataset transparency and responsible use, we release our entire multimodal
  audit, allowing practitioners to trace data provenance across text, speech, and
  video.
paper_ai_keywords: multimodal machine learning applications web-crawled synthetic social media platforms YouTube dataset derivations restrictively licensed non-commercial restrictions geographical representation multilingual representation Western-centricity data sourcing dataset transparency responsible use data provenance