title: Consent in Crisis: The Rapid Decline of the AI Data Commons
summary: General-purpose artificial intelligence (AI) systems are built on massive
  swathes of public web data, assembled into corpora such as C4, RefinedWeb, and
  Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit
  of the consent protocols for the web domains underlying AI training corpora.
  Our audit of 14,000 web domains provides an expansive view of crawlable web
  data and how consent preferences to use it are changing over time. We observe a
  proliferation of AI-specific clauses to limit use, acute differences in
  restrictions on AI developers, as well as general inconsistencies between
  websites' expressed intentions in their Terms of Service and their robots.txt.
  We diagnose these as symptoms of ineffective web protocols, not designed to
  cope with the widespread re-purposing of the internet for AI. Our longitudinal
  analyses show that in a single year (2023-2024) there has been a rapid
  crescendo of data restrictions from web sources, rendering ~5%+ of all tokens
  in C4, or 28%+ of the most actively maintained, critical sources in C4, fully
  restricted from use. For Terms of Service crawling restrictions, a full 45% of
  C4 is now restricted. If respected or enforced, these restrictions are rapidly
  biasing the diversity, freshness, and scaling laws for general-purpose AI
  systems. We hope to illustrate the emerging crisis in data consent, foreclosing
  much of the open web, not only for commercial AI, but non-commercial AI and
  academic purposes.
paper_authors._id: 669f84f13e1b77c317b64345
paper_authors.name: Shayne Longpre
paper_authors._id: 669f84f13e1b77c317b64346
paper_authors.name: Robert Mahari
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:35:25.946Z
paper_authors._id: 650e085c37698373cdda7a6d
paper_authors.avatarUrl: /avatars/8fae18709d51dd86aa6eddcfb2abeae8.svg
paper_authors.fullname: Robert Mahari
paper_authors.type: user
paper_authors.user: rmahari
paper_authors._id: 669f84f13e1b77c317b64347
paper_authors.name: Ariel Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-29T07:06:14.077Z
paper_authors._id: 638bcfa91987d67b340e6c1c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638bcfa91987d67b340e6c1c/3tHCB_J6c4-lsEZ_zJSlp.jpeg
paper_authors.fullname: Ariel N. Lee
paper_authors.type: user
paper_authors.user: arielnlee
paper_authors._id: 669f84f13e1b77c317b64348
paper_authors.name: Campbell Lund
paper_authors._id: 669f84f13e1b77c317b64349
paper_authors.name: Hamidah Oderinwale
paper_authors._id: 669f84f13e1b77c317b6434a
paper_authors.name: William Brannon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:46:19.380Z
paper_authors._id: 641bdd25a63c4e8062387b6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641bdd25a63c4e8062387b6a/dFanBhG6_NVqpRNB-jjC4.png
paper_authors.fullname: William Brannon
paper_authors.type: user
paper_authors.user: wwbrannon
paper_authors._id: 669f84f13e1b77c317b6434b
paper_authors.name: Nayan Saxena
paper_authors._id: 669f84f13e1b77c317b6434c
paper_authors.name: Naana Obeng-Marnu
paper_authors._id: 669f84f13e1b77c317b6434d
paper_authors.name: Tobin South
paper_authors._id: 669f84f13e1b77c317b6434e
paper_authors.name: Cole Hunter
paper_authors._id: 669f84f13e1b77c317b6434f
paper_authors.name: Kevin Klyman
paper_authors._id: 669f84f13e1b77c317b64350
paper_authors.name: Christopher Klamm
paper_authors._id: 669f84f13e1b77c317b64351
paper_authors.name: Hailey Schoelkopf
paper_authors._id: 669f84f13e1b77c317b64352
paper_authors.name: Nikhil Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:47:45.558Z
paper_authors._id: 6374384c250cf1379bc278f8
paper_authors.avatarUrl: /avatars/e436988005ad4f9d99fcafb8bdfade9c.svg
paper_authors.fullname: Nikhil Singh
paper_authors.type: user
paper_authors.user: nsingh1
paper_authors._id: 669f84f13e1b77c317b64353
paper_authors.name: Manuel Cherep
paper_authors._id: 669f84f13e1b77c317b64354
paper_authors.name: Ahmad Anis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:08:10.262Z
paper_authors._id: 6246908d8031dcfa9ef6d80b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6246908d8031dcfa9ef6d80b/hVdURjUl1RS2MZf4qOhvI.jpeg
paper_authors.fullname: Ahmad Mustafa Anis
paper_authors.type: user
paper_authors.user: AhmadMustafa
paper_authors._id: 669f84f13e1b77c317b64355
paper_authors.name: An Dinh
paper_authors._id: 669f84f13e1b77c317b64356
paper_authors.name: Caroline Chitongo
paper_authors._id: 669f84f13e1b77c317b64357
paper_authors.name: Da Yin
paper_authors._id: 669f84f13e1b77c317b64358
paper_authors.name: Damien Sileo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-29T07:06:15.859Z
paper_authors._id: 5fc0bcb41160c47d1d43856b
paper_authors.avatarUrl: /avatars/4d68767d363f7bc5605906c0e994f132.svg
paper_authors.fullname: Damien Sileo
paper_authors.type: user
paper_authors.user: sileod
paper_authors._id: 669f84f13e1b77c317b64359
paper_authors.name: Deividas Mataciunas
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-29T14:58:08.268Z
paper_authors._id: 6040a00558b78f3a0047c23a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6040a00558b78f3a0047c23a/_BsyJoaCBO3r6GnfSgnIA.jpeg
paper_authors.fullname: David Mataciunas
paper_authors.type: user
paper_authors.user: DeividasM
paper_authors._id: 669f84f13e1b77c317b6435a
paper_authors.name: Diganta Misra
paper_authors._id: 669f84f13e1b77c317b6435b
paper_authors.name: Emad Alghamdi
paper_authors._id: 669f84f13e1b77c317b6435c
paper_authors.name: Enrico Shippole
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T07:27:13.474Z
paper_authors._id: 6276ba3c2d26ac639e5a2b01
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6276ba3c2d26ac639e5a2b01/ULDyT9ijOye7rfXqTWFVG.jpeg
paper_authors.fullname: Enrico Shippole
paper_authors.type: user
paper_authors.user: conceptofmind
paper_authors._id: 669f84f13e1b77c317b6435d
paper_authors.name: Jianguo Zhang
paper_authors._id: 669f84f13e1b77c317b6435e
paper_authors.name: Joanna Materzynska
paper_authors._id: 669f84f13e1b77c317b6435f
paper_authors.name: Kun Qian
paper_authors._id: 669f84f13e1b77c317b64360
paper_authors.name: Kush Tiwary
paper_authors._id: 669f84f13e1b77c317b64361
paper_authors.name: Lester Miranda
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-13T07:39:45.367Z
paper_authors._id: 634e20a0c1ce28f1de920cc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
paper_authors.fullname: Lj V. Miranda
paper_authors.type: user
paper_authors.user: ljvmiranda921
paper_authors._id: 669f84f13e1b77c317b64362
paper_authors.name: Manan Dey
paper_authors._id: 669f84f13e1b77c317b64363
paper_authors.name: Minnie Liang
paper_authors._id: 669f84f13e1b77c317b64364
paper_authors.name: Mohammed Hamdy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T11:43:16.696Z
paper_authors._id: 62645f88c39850dc093d6105
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650745211725-noauth.png
paper_authors.fullname: Mohammed Hamdy
paper_authors.type: user
paper_authors.user: mmhamdy
paper_authors._id: 669f84f13e1b77c317b64365
paper_authors.name: Niklas Muennighoff
paper_authors._id: 669f84f13e1b77c317b64366
paper_authors.name: Seonghyeon Ye
paper_authors._id: 669f84f13e1b77c317b64367
paper_authors.name: Seungone Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-02T07:19:11.856Z
paper_authors._id: 6469949654873f0043b09c22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
paper_authors.fullname: Seungone Kim
paper_authors.type: user
paper_authors.user: seungone
paper_authors._id: 669f84f13e1b77c317b64368
paper_authors.name: Shrestha Mohanty
paper_authors._id: 669f84f13e1b77c317b64369
paper_authors.name: Vipul Gupta
paper_authors._id: 669f84f13e1b77c317b6436a
paper_authors.name: Vivek Sharma
paper_authors._id: 669f84f13e1b77c317b6436b
paper_authors.name: Vu Minh Chien
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-26T07:40:55.488Z
paper_authors._id: 60535c9d10aba34e3b6a2ef7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1616075850230-noauth.jpeg
paper_authors.fullname: vumichien
paper_authors.type: user
paper_authors.user: vumichien
paper_authors._id: 669f84f13e1b77c317b6436c
paper_authors.name: Xuhui Zhou
paper_authors._id: 669f84f13e1b77c317b6436d
paper_authors.name: Yizhi Li
paper_authors._id: 669f84f13e1b77c317b6436e
paper_authors.name: Caiming Xiong
paper_authors._id: 669f84f13e1b77c317b6436f
paper_authors.name: Luis Villa
paper_authors._id: 669f84f13e1b77c317b64370
paper_authors.name: Stella Biderman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:51:07.896Z
paper_authors._id: 60347d3660e3dd96631c9093
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
paper_authors.fullname: Stella Biderman
paper_authors.type: user
paper_authors.user: stellaathena
paper_authors._id: 669f84f13e1b77c317b64371
paper_authors.name: Hanlin Li
paper_authors._id: 669f84f13e1b77c317b64372
paper_authors.name: Daphne Ippolito
paper_authors._id: 669f84f13e1b77c317b64373
paper_authors.name: Sara Hooker
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T07:48:51.508Z
paper_authors._id: 63434eb76f59b79da07dbddf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
paper_authors.fullname: Sara Hooker
paper_authors.type: user
paper_authors.user: sarahooker
paper_authors._id: 669f84f13e1b77c317b64374
paper_authors.name: Jad Kabbara
paper_authors._id: 669f84f13e1b77c317b64375
paper_authors.name: Sandy Pentland
paper_title: Consent in Crisis: The Rapid Decline of the AI Data Commons
paper_summary: General-purpose artificial intelligence (AI) systems are built on massive
  swathes of public web data, assembled into corpora such as C4, RefinedWeb, and
  Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit
  of the consent protocols for the web domains underlying AI training corpora.
  Our audit of 14,000 web domains provides an expansive view of crawlable web
  data and how consent preferences to use it are changing over time. We observe a
  proliferation of AI-specific clauses to limit use, acute differences in
  restrictions on AI developers, as well as general inconsistencies between
  websites' expressed intentions in their Terms of Service and their robots.txt.
  We diagnose these as symptoms of ineffective web protocols, not designed to
  cope with the widespread re-purposing of the internet for AI. Our longitudinal
  analyses show that in a single year (2023-2024) there has been a rapid
  crescendo of data restrictions from web sources, rendering ~5%+ of all tokens
  in C4, or 28%+ of the most actively maintained, critical sources in C4, fully
  restricted from use. For Terms of Service crawling restrictions, a full 45% of
  C4 is now restricted. If respected or enforced, these restrictions are rapidly
  biasing the diversity, freshness, and scaling laws for general-purpose AI
  systems. We hope to illustrate the emerging crisis in data consent, foreclosing
  much of the open web, not only for commercial AI, but non-commercial AI and
  academic purposes.
paper_ai_summary: The audit of web domains used for training AI systems reveals a growing inconsistency and restriction of data use, posing a significant threat to the diversity and availability of open web data for AI.
paper_ai_keywords: Terms of Service robots.txt data restrictions AI training corpora consent protocols data consent crisis