title: A Framework for Automated Measurement of Responsible AI Harms in
    Generative AI Applications
summary: We present a framework for the automated measurement of responsible AI (RAI)
  metrics for large language models (LLMs) and associated products and services.
  Our framework for automatically measuring harms from LLMs builds on existing
  technical and sociotechnical expertise and leverages the capabilities of
  state-of-the-art LLMs, such as GPT-4. We use this framework to run through
  several case studies investigating how different LLMs may violate a range of
  RAI-related principles. The framework may be employed alongside domain-specific
  sociotechnical expertise to create measurements for new harm areas in the
  future. By implementing this framework, we aim to enable more advanced harm
  measurement efforts and further the responsible use of LLMs.
paper_authors._id: 653f0e74e337eff8f51e3a89
paper_authors.name: Ahmed Magooda
paper_authors._id: 653f0e74e337eff8f51e3a8a
paper_authors.name: Alec Helyar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:57:08.521Z
paper_authors._id: 6385710c20e00df40e4e5275
paper_authors.avatarUrl: /avatars/bd793f0f456cbb1fca47ce9aca9c2acc.svg
paper_authors.fullname: Alec Helyar
paper_authors.type: user
paper_authors.user: alech97
paper_authors._id: 653f0e74e337eff8f51e3a8b
paper_authors.name: Kyle Jackson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:57:59.197Z
paper_authors._id: 64a70f96fa4e39a4a653886c
paper_authors.avatarUrl: /avatars/7e3fd4399cdf75bc9f01cf7b31db9fda.svg
paper_authors.fullname: Kyle Jackson
paper_authors.type: user
paper_authors.user: kjacks
paper_authors._id: 653f0e74e337eff8f51e3a8c
paper_authors.name: David Sullivan
paper_authors._id: 653f0e74e337eff8f51e3a8d
paper_authors.name: Chad Atalla
paper_authors._id: 653f0e74e337eff8f51e3a8e
paper_authors.name: Emily Sheng
paper_authors._id: 653f0e74e337eff8f51e3a8f
paper_authors.name: Dan Vann
paper_authors._id: 653f0e74e337eff8f51e3a90
paper_authors.name: Richard Edgar
paper_authors._id: 653f0e74e337eff8f51e3a91
paper_authors.name: Hamid Palangi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:59:04.573Z
paper_authors._id: 6189ddd0992df2640e3e7d40
paper_authors.avatarUrl: /avatars/925f2308fc412ae352b57a1b71815028.svg
paper_authors.fullname: Hamid Palangi
paper_authors.type: user
paper_authors.user: hamidpalangi
paper_authors._id: 653f0e74e337eff8f51e3a92
paper_authors.name: Roman Lutz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:59:14.107Z
paper_authors._id: 6429db4169bb3e94e9a28de7
paper_authors.avatarUrl: /avatars/4e0ecef3f9f2fe0289de063e37d4dd68.svg
paper_authors.fullname: Roman Lutz
paper_authors.type: user
paper_authors.user: romanlutz
paper_authors._id: 653f0e74e337eff8f51e3a93
paper_authors.name: Hongliang Kong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:59:27.119Z
paper_authors._id: 634ae8541fd4a53b672a3a77
paper_authors.avatarUrl: /avatars/fff40c0e090dafc5e9ee566b04b1069d.svg
paper_authors.fullname: Kong
paper_authors.type: user
paper_authors.user: Hongliang
paper_authors._id: 653f0e74e337eff8f51e3a94
paper_authors.name: Vincent Yun
paper_authors._id: 653f0e74e337eff8f51e3a95
paper_authors.name: Eslam Kamal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:59:45.625Z
paper_authors._id: 63aa0841769a10efc400dc6b
paper_authors.avatarUrl: /avatars/5add339a8006b2e827b8b2de03a60912.svg
paper_authors.fullname: ka
paper_authors.type: user
paper_authors.user: eslamkamal22
paper_authors._id: 653f0e74e337eff8f51e3a96
paper_authors.name: Federico Zarfati
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:59:55.804Z
paper_authors._id: 64c6d0a2568845d756442cb8
paper_authors.avatarUrl: /avatars/90fade8b9cfc806ffb0595f5520cc62a.svg
paper_authors.fullname: Federico Zarfati
paper_authors.type: user
paper_authors.user: onitaku
paper_authors._id: 653f0e74e337eff8f51e3a97
paper_authors.name: Hanna Wallach
paper_authors._id: 653f0e74e337eff8f51e3a98
paper_authors.name: Sarah Bird
paper_authors._id: 653f0e74e337eff8f51e3a99
paper_authors.name: Mei Chen
paper_title: A Framework for Automated Measurement of Responsible AI Harms in
    Generative AI Applications
paper_summary: We present a framework for the automated measurement of responsible AI (RAI)
  metrics for large language models (LLMs) and associated products and services.
  Our framework for automatically measuring harms from LLMs builds on existing
  technical and sociotechnical expertise and leverages the capabilities of
  state-of-the-art LLMs, such as GPT-4. We use this framework to run through
  several case studies investigating how different LLMs may violate a range of
  RAI-related principles. The framework may be employed alongside domain-specific
  sociotechnical expertise to create measurements for new harm areas in the
  future. By implementing this framework, we aim to enable more advanced harm
  measurement efforts and further the responsible use of LLMs.
paper_ai_summary: A framework measures responsible AI metrics for large language models to assess and mitigate harm, leveraging existing expertise and advanced language model capabilities.
paper_ai_keywords: large language models LLMs responsible AI RAI GPT-4 sociotechnical expertise harm measurement