title: Evaluating the Social Impact of Generative AI Systems in Systems and
    Society
summary: Generative AI systems across modalities, ranging from text, image, audio, and
  video, have broad social impacts, but there exists no official standard for
  means of evaluating those impacts and which impacts should be evaluated. We
  move toward a standard approach in evaluating a generative AI system for any
  modality, in two overarching categories: what is able to be evaluated in a base
  system that has no predetermined application and what is able to be evaluated
  in society. We describe specific social impact categories and how to approach
  and conduct evaluations in the base technical system, then in people and
  society. Our framework for a base system defines seven categories of social
  impact: bias, stereotypes, and representational harms; cultural values and
  sensitive content; disparate performance; privacy and data protection;
  financial costs; environmental costs; and data and content moderation labor
  costs. Suggested methods for evaluation apply to all modalities and analyses of
  the limitations of existing evaluations serve as a starting point for necessary
  investment in future evaluations. We offer five overarching categories for what
  is able to be evaluated in society, each with their own subcategories:
  trustworthiness and autonomy; inequality, marginalization, and violence;
  concentration of authority; labor and creativity; and ecosystem and
  environment. Each subcategory includes recommendations for mitigating harm. We
  are concurrently crafting an evaluation repository for the AI research
  community to contribute existing evaluations along the given categories. This
  version will be updated following a CRAFT session at ACM FAccT 2023.
paper_authors._id: 6486818b43ba3df95402d2cf
paper_authors.name: Irene Solaiman
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-12T04:26:55.608Z
paper_authors._id: 62543749b777cd32720675c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658760912583-62543749b777cd32720675c2.jpeg
paper_authors.fullname: Irene Solaiman
paper_authors.type: user
paper_authors.user: irenesolaiman
paper_authors._id: 6486818b43ba3df95402d2d0
paper_authors.name: Zeerak Talat
paper_authors._id: 6486818b43ba3df95402d2d1
paper_authors.name: William Agnew
paper_authors._id: 6486818b43ba3df95402d2d2
paper_authors.name: Lama Ahmad
paper_authors._id: 6486818b43ba3df95402d2d3
paper_authors.name: Dylan Baker
paper_authors._id: 6486818b43ba3df95402d2d4
paper_authors.name: Su Lin Blodgett
paper_authors._id: 6486818b43ba3df95402d2d5
paper_authors.name: Hal Daumé III
paper_authors._id: 6486818b43ba3df95402d2d6
paper_authors.name: Jesse Dodge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T10:09:47.178Z
paper_authors._id: 6283f38567d336d3e5d5280e
paper_authors.avatarUrl: /avatars/d0a54aaec74a90b050e671c191b87a80.svg
paper_authors.fullname: Jesse Dodge
paper_authors.type: user
paper_authors.user: JesseDodge
paper_authors._id: 6486818b43ba3df95402d2d7
paper_authors.name: Ellie Evans
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T10:10:45.057Z
paper_authors._id: 626c0912974d6a67df700db6
paper_authors.avatarUrl: /avatars/057a19f682e6a8d0411bc6efa25f2588.svg
paper_authors.fullname: Ellie Evans
paper_authors.type: user
paper_authors.user: ellieevans
paper_authors._id: 6486818b43ba3df95402d2d8
paper_authors.name: Sara Hooker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T10:11:04.380Z
paper_authors._id: 63434eb76f59b79da07dbddf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
paper_authors.fullname: Sara Hooker
paper_authors.type: user
paper_authors.user: sarahooker
paper_authors._id: 6486818b43ba3df95402d2d9
paper_authors.name: Yacine Jernite
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T10:07:51.393Z
paper_authors._id: 5ee3a7cd2a3eae3cbdad1305
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594144055859-5ee3a7cd2a3eae3cbdad1305.jpeg
paper_authors.fullname: Yacine Jernite
paper_authors.type: user
paper_authors.user: yjernite
paper_authors._id: 6486818b43ba3df95402d2da
paper_authors.name: Alexandra Sasha Luccioni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T10:08:05.485Z
paper_authors._id: 60edd0133e2c73a9a21455f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60edd0133e2c73a9a21455f5/yK1G-Fv-YjYb7v_chkz3p.jpeg
paper_authors.fullname: Sasha Luccioni
paper_authors.type: user
paper_authors.user: sasha
paper_authors._id: 6486818b43ba3df95402d2db
paper_authors.name: Alberto Lusoli
paper_authors._id: 6486818b43ba3df95402d2dc
paper_authors.name: Margaret Mitchell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T10:08:18.780Z
paper_authors._id: 60c757ea5f9a76ab3f844f12
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1626214544196-60c757ea5f9a76ab3f844f12.png
paper_authors.fullname: Margaret Mitchell
paper_authors.type: user
paper_authors.user: meg
paper_authors._id: 6486818b43ba3df95402d2dd
paper_authors.name: Jessica Newman
paper_authors._id: 6486818b43ba3df95402d2de
paper_authors.name: Marie-Therese Png
paper_authors._id: 6486818b43ba3df95402d2df
paper_authors.name: Andrew Strait
paper_authors._id: 6486818b43ba3df95402d2e0
paper_authors.name: Apostol Vassilev
paper_title: Evaluating the Social Impact of Generative AI Systems in Systems and
    Society
paper_summary: Generative AI systems across modalities, ranging from text, image, audio, and
  video, have broad social impacts, but there exists no official standard for
  means of evaluating those impacts and which impacts should be evaluated. We
  move toward a standard approach in evaluating a generative AI system for any
  modality, in two overarching categories: what is able to be evaluated in a base
  system that has no predetermined application and what is able to be evaluated
  in society. We describe specific social impact categories and how to approach
  and conduct evaluations in the base technical system, then in people and
  society. Our framework for a base system defines seven categories of social
  impact: bias, stereotypes, and representational harms; cultural values and
  sensitive content; disparate performance; privacy and data protection;
  financial costs; environmental costs; and data and content moderation labor
  costs. Suggested methods for evaluation apply to all modalities and analyses of
  the limitations of existing evaluations serve as a starting point for necessary
  investment in future evaluations. We offer five overarching categories for what
  is able to be evaluated in society, each with their own subcategories:
  trustworthiness and autonomy; inequality, marginalization, and violence;
  concentration of authority; labor and creativity; and ecosystem and
  environment. Each subcategory includes recommendations for mitigating harm. We
  are concurrently crafting an evaluation repository for the AI research
  community to contribute existing evaluations along the given categories. This
  version will be updated following a CRAFT session at ACM FAccT 2023.