title: Introducing v0.5 of the AI Safety Benchmark from MLCommons
summary: This paper introduces v0.5 of the AI Safety Benchmark, which has been created
  by the MLCommons AI Safety Working Group. The AI Safety Benchmark has been
  designed to assess the safety risks of AI systems that use chat-tuned language
  models. We introduce a principled approach to specifying and constructing the
  benchmark, which for v0.5 covers only a single use case (an adult chatting to a
  general-purpose assistant in English), and a limited set of personas (i.e.,
  typical users, malicious users, and vulnerable users). We created a new
  taxonomy of 13 hazard categories, of which 7 have tests in the v0.5 benchmark.
  We plan to release version 1.0 of the AI Safety Benchmark by the end of 2024.
  The v1.0 benchmark will provide meaningful insights into the safety of AI
  systems. However, the v0.5 benchmark should not be used to assess the safety of
  AI systems. We have sought to fully document the limitations, flaws, and
  challenges of v0.5. This release of v0.5 of the AI Safety Benchmark includes
  (1) a principled approach to specifying and constructing the benchmark, which
  comprises use cases, types of systems under test (SUTs), language and context,
  personas, tests, and test items; (2) a taxonomy of 13 hazard categories with
  definitions and subcategories; (3) tests for seven of the hazard categories,
  each comprising a unique set of test items, i.e., prompts. There are 43,090
  test items in total, which we created with templates; (4) a grading system for
  AI systems against the benchmark; (5) an openly available platform, and
  downloadable tool, called ModelBench that can be used to evaluate the safety of
  AI systems on the benchmark; (6) an example evaluation report which benchmarks
  the performance of over a dozen openly available chat-tuned language models;
  (7) a test specification for the benchmark.
paper_authors._id: 6621ced43ec06d6503eecd9e
paper_authors.name: Bertie Vidgen
paper_authors._id: 6621ced43ec06d6503eecd9f
paper_authors.name: Adarsh Agrawal
paper_authors._id: 6621ced43ec06d6503eecda0
paper_authors.name: Ahmed M. Ahmed
paper_authors._id: 6621ced43ec06d6503eecda1
paper_authors.name: Victor Akinwande
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:54:25.019Z
paper_authors._id: 643b07dcb54b346556532841
paper_authors.avatarUrl: /avatars/7806b22b86a22306d5944e75a822787f.svg
paper_authors.fullname: Victor Akinwande
paper_authors.type: user
paper_authors.user: vakinwan
paper_authors._id: 6621ced43ec06d6503eecda2
paper_authors.name: Namir Al-Nuaimi
paper_authors._id: 6621ced43ec06d6503eecda3
paper_authors.name: Najla Alfaraj
paper_authors._id: 6621ced43ec06d6503eecda4
paper_authors.name: Elie Alhajjar
paper_authors._id: 6621ced43ec06d6503eecda5
paper_authors.name: Lora Aroyo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:54:42.242Z
paper_authors._id: 659d7258dde639191fa89a34
paper_authors.avatarUrl: /avatars/ccbe525e7444aba27b7f4e1d91553c65.svg
paper_authors.fullname: Lora Aroyo
paper_authors.type: user
paper_authors.user: laroyo
paper_authors._id: 6621ced43ec06d6503eecda6
paper_authors.name: Trupti Bavalatti
paper_authors._id: 6621ced43ec06d6503eecda7
paper_authors.name: Borhane Blili-Hamelin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:54:52.910Z
paper_authors._id: 64149f87634b1a9b24cd237b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679073077831-noauth.jpeg
paper_authors.fullname: Borhane Blili-Hamelin
paper_authors.type: user
paper_authors.user: borhane
paper_authors._id: 6621ced43ec06d6503eecda8
paper_authors.name: Kurt Bollacker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:54:59.655Z
paper_authors._id: 6570a4139c503c4a6af43279
paper_authors.avatarUrl: /avatars/315c4bf8c5acd44364c8f899c40d6a02.svg
paper_authors.fullname: Kurt Bollacker
paper_authors.type: user
paper_authors.user: bollacker
paper_authors._id: 6621ced43ec06d6503eecda9
paper_authors.name: Rishi Bomassani
paper_authors._id: 6621ced43ec06d6503eecdaa
paper_authors.name: Marisa Ferrara Boston
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T20:46:31.914Z
paper_authors._id: 65303a00d849382d5559a67e
paper_authors.avatarUrl: /avatars/a599ca55c7aeee37f4298272cfcb064d.svg
paper_authors.fullname: Marisa
paper_authors.type: user
paper_authors.user: maReins
paper_authors._id: 6621ced43ec06d6503eecdab
paper_authors.name: Siméon Campos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:55:20.028Z
paper_authors._id: 61740ed8ab394c394536a082
paper_authors.avatarUrl: /avatars/bed4740530478b643440b8725b001340.svg
paper_authors.fullname: Simeon Campos
paper_authors.type: user
paper_authors.user: WayZNLP
paper_authors._id: 6621ced43ec06d6503eecdac
paper_authors.name: Kal Chakra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:55:26.523Z
paper_authors._id: 64e0666a8e2084e1d7c77c19
paper_authors.avatarUrl: /avatars/3e0a9c71f0f0ed9315ccaef46625fbfc.svg
paper_authors.fullname: Kal Chakra
paper_authors.type: user
paper_authors.user: kalchakra
paper_authors._id: 6621ced43ec06d6503eecdad
paper_authors.name: Canyu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:55:33.404Z
paper_authors._id: 6483af58571c2dcfa98cae82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6483af58571c2dcfa98cae82/d8Vnh_EG1KxeEoNAAQXHZ.jpeg
paper_authors.fullname: Canyu Chen
paper_authors.type: user
paper_authors.user: canyuchen
paper_authors._id: 6621ced43ec06d6503eecdae
paper_authors.name: Cody Coleman
paper_authors._id: 6621ced43ec06d6503eecdaf
paper_authors.name: Zacharie Delpierre Coudert
paper_authors._id: 6621ced43ec06d6503eecdb0
paper_authors.name: Leon Derczynski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:55:54.536Z
paper_authors._id: 61caf9d13918092781f7b745
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1640692164586-noauth.jpeg
paper_authors.fullname: Leon Derczynski
paper_authors.type: user
paper_authors.user: leondz
paper_authors._id: 6621ced43ec06d6503eecdb1
paper_authors.name: Debojyoti Dutta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:56:06.895Z
paper_authors._id: 6191539da26ced2814661ec8
paper_authors.avatarUrl: /avatars/ac5fd0906704549d69a72834866dc8fc.svg
paper_authors.fullname: debojyoti dutta
paper_authors.type: user
paper_authors.user: ddutta
paper_authors._id: 6621ced43ec06d6503eecdb2
paper_authors.name: Ian Eisenberg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:56:12.889Z
paper_authors._id: 62a954409f60be260fbfa5b9
paper_authors.avatarUrl: /avatars/22f914656ba9c840b32812bca7104a17.svg
paper_authors.fullname: Ian Eisenberg
paper_authors.type: user
paper_authors.user: IanE
paper_authors._id: 6621ced43ec06d6503eecdb3
paper_authors.name: James Ezick
paper_authors._id: 6621ced43ec06d6503eecdb4
paper_authors.name: Heather Frase
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:56:22.184Z
paper_authors._id: 647dd166e4d52fe0e01cb11b
paper_authors.avatarUrl: /avatars/934ac7ecacd607585585bbcbaa991143.svg
paper_authors.fullname: Heather Frase
paper_authors.type: user
paper_authors.user: Hnfrase
paper_authors._id: 6621ced43ec06d6503eecdb5
paper_authors.name: Brian Fuller
paper_authors._id: 6621ced43ec06d6503eecdb6
paper_authors.name: Ram Gandikota
paper_authors._id: 6621ced43ec06d6503eecdb7
paper_authors.name: Agasthya Gangavarapu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:56:53.552Z
paper_authors._id: 64ed26cf476cdc850285d596
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GvS4Z1kAuocgy385AnL6n.jpeg
paper_authors.fullname: Agasthya Gangavarapu
paper_authors.type: user
paper_authors.user: itzAugust
paper_authors._id: 6621ced43ec06d6503eecdb8
paper_authors.name: Ananya Gangavarapu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:57:01.200Z
paper_authors._id: 64e3e777d5e75575cd79fe59
paper_authors.avatarUrl: /avatars/091ed4cfdf8442b06800580872684e1e.svg
paper_authors.fullname: Ananya Gangavarapu
paper_authors.type: user
paper_authors.user: anigasan
paper_authors._id: 6621ced43ec06d6503eecdb9
paper_authors.name: James Gealy
paper_authors._id: 6621ced43ec06d6503eecdba
paper_authors.name: Rajat Ghosh
paper_authors._id: 6621ced43ec06d6503eecdbb
paper_authors.name: James Goel
paper_authors._id: 6621ced43ec06d6503eecdbc
paper_authors.name: Usman Gohar
paper_authors._id: 6621ced43ec06d6503eecdbd
paper_authors.name: Sujata Goswami
paper_authors._id: 6621ced43ec06d6503eecdbe
paper_authors.name: Scott A. Hale
paper_authors._id: 6621ced43ec06d6503eecdbf
paper_authors.name: Wiebke Hutiri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:06:10.711Z
paper_authors._id: 640aff86b871d7117d612d1e
paper_authors.avatarUrl: /avatars/740e7328e379e84d48b70a4230f9b8aa.svg
paper_authors.fullname: Wiebke Hutiri
paper_authors.type: user
paper_authors.user: wiebke-codes
paper_authors._id: 6621ced43ec06d6503eecdc0
paper_authors.name: Joseph Marvin Imperial
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-21T07:24:36.416Z
paper_authors._id: 620a1e42e190de4b8f27ba1f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644830238640-noauth.jpeg
paper_authors.fullname: Joseph Imperial
paper_authors.type: user
paper_authors.user: josephimperial
paper_authors._id: 6621ced43ec06d6503eecdc1
paper_authors.name: Surgan Jandial
paper_authors._id: 6621ced43ec06d6503eecdc2
paper_authors.name: Nick Judd
paper_authors._id: 6621ced43ec06d6503eecdc3
paper_authors.name: Felix Juefei-Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T16:55:51.175Z
paper_authors._id: 6417cf37dce1e4c0229f17b1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6417cf37dce1e4c0229f17b1/7h-ZCB5f4wif7TsnF-B1M.jpeg
paper_authors.fullname: Felix Xu
paper_authors.type: user
paper_authors.user: katanaxu
paper_authors._id: 6621ced43ec06d6503eecdc4
paper_authors.name: Foutse Khomh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:05:47.554Z
paper_authors._id: 65d60476876b7d939870c5af
paper_authors.avatarUrl: /avatars/57a35d093d25ac084ebffcaf9e3599ee.svg
paper_authors.fullname: Foutse Khomh
paper_authors.type: user
paper_authors.user: foutsekh
paper_authors._id: 6621ced43ec06d6503eecdc5
paper_authors.name: Bhavya Kailkhura
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:05:54.753Z
paper_authors._id: 65cb79db6427380bc21261e2
paper_authors.avatarUrl: /avatars/a003eb5d0955417329c1a4170ae65879.svg
paper_authors.fullname: Bhavya Kailkhura
paper_authors.type: user
paper_authors.user: bhavyakailkhura
paper_authors._id: 6621ced43ec06d6503eecdc6
paper_authors.name: Hannah Rose Kirk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:06:01.953Z
paper_authors._id: 6255baa66b1f65958caf026c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649842998853-6255baa66b1f65958caf026c.jpeg
paper_authors.fullname: Hannah Rose Kirk
paper_authors.type: user
paper_authors.user: HannahRoseKirk
paper_authors._id: 6621ced43ec06d6503eecdc7
paper_authors.name: Kevin Klyman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:05:11.282Z
paper_authors._id: 64817160ec65b8b77d83e910
paper_authors.avatarUrl: /avatars/0dc892cccb8c71a1ad1c1a1801ee9772.svg
paper_authors.fullname: Kevin Klyman
paper_authors.type: user
paper_authors.user: kklyman
paper_authors._id: 6621ced43ec06d6503eecdc8
paper_authors.name: Chris Knotz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:05:04.418Z
paper_authors._id: 65f8aeebc1823e23cf92aa97
paper_authors.avatarUrl: /avatars/4a3a81d466bd194945b1b59dcdfbdad9.svg
paper_authors.fullname: Chris Knotz
paper_authors.type: user
paper_authors.user: ChrisKnotz
paper_authors._id: 6621ced43ec06d6503eecdc9
paper_authors.name: Michael Kuchnik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:04:57.028Z
paper_authors._id: 63bc3dad1374e3ef9131d75c
paper_authors.avatarUrl: /avatars/d58f700ac56b6bf11e18aaf5f8d2691c.svg
paper_authors.fullname: Michael Kuchnik
paper_authors.type: user
paper_authors.user: mkuchnik
paper_authors._id: 6621ced43ec06d6503eecdca
paper_authors.name: Shachi H. Kumar
paper_authors._id: 6621ced43ec06d6503eecdcb
paper_authors.name: Chris Lengerich
paper_authors._id: 6621ced43ec06d6503eecdcc
paper_authors.name: Bo Li
paper_authors._id: 6621ced43ec06d6503eecdcd
paper_authors.name: Zeyi Liao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:59:20.810Z
paper_authors._id: 632ed3e534c562cb1d328fa7
paper_authors.avatarUrl: /avatars/e29e47e734740787ea40c729eec146b5.svg
paper_authors.fullname: zeyi liao
paper_authors.type: user
paper_authors.user: lzy337
paper_authors._id: 6621ced43ec06d6503eecdce
paper_authors.name: Eileen Peters Long
paper_authors._id: 6621ced43ec06d6503eecdcf
paper_authors.name: Victor Lu
paper_authors._id: 6621ced43ec06d6503eecdd0
paper_authors.name: Yifan Mai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:05:19.831Z
paper_authors._id: 645beb41f9d4ec91fdd935d9
paper_authors.avatarUrl: /avatars/a4bddecae3096088613b915b42354b81.svg
paper_authors.fullname: Yifan Mai
paper_authors.type: user
paper_authors.user: yifanmai
paper_authors._id: 6621ced43ec06d6503eecdd1
paper_authors.name: Priyanka Mary Mammen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:05:27.381Z
paper_authors._id: 66036ac2fb66ef4a5cf82fe5
paper_authors.avatarUrl: /avatars/d9e9bd3b5e651453b3bc76fbfdf8cf97.svg
paper_authors.fullname: Priyanka Mary Mammen
paper_authors.type: user
paper_authors.user: pmammen
paper_authors._id: 6621ced43ec06d6503eecdd2
paper_authors.name: Kelvin Manyeki
paper_authors._id: 6621ced43ec06d6503eecdd3
paper_authors.name: Sean McGregor
paper_authors._id: 6621ced43ec06d6503eecdd4
paper_authors.name: Virendra Mehta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-23T13:07:16.939Z
paper_authors._id: 604eb19e3050a33ebb17ef58
paper_authors.avatarUrl: /avatars/23650002ba3befee83060fe978a251c8.svg
paper_authors.fullname: Virendra Mehta
paper_authors.type: user
paper_authors.user: Veeru
paper_authors._id: 6621ced43ec06d6503eecdd5
paper_authors.name: Shafee Mohammed
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:03:50.738Z
paper_authors._id: 62bf8e6be140faaa8532a128
paper_authors.avatarUrl: /avatars/c0a442a952f372d33b9f517cfad36500.svg
paper_authors.fullname: shafee mohammed
paper_authors.type: user
paper_authors.user: shafcodes
paper_authors._id: 6621ced43ec06d6503eecdd6
paper_authors.name: Emanuel Moss
paper_authors._id: 6621ced43ec06d6503eecdd7
paper_authors.name: Lama Nachman
paper_authors._id: 6621ced43ec06d6503eecdd8
paper_authors.name: Dinesh Jinenhally Naganna
paper_authors._id: 6621ced43ec06d6503eecdd9
paper_authors.name: Amin Nikanjam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:02:46.503Z
paper_authors._id: 65d754e96f1ee3a269bfa30d
paper_authors.avatarUrl: /avatars/4d579a4a23ef8618b929a50ca4c172fb.svg
paper_authors.fullname: Amin Nikanjam
paper_authors.type: user
paper_authors.user: aminnikanjam
paper_authors._id: 6621ced43ec06d6503eecdda
paper_authors.name: Besmira Nushi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:02:53.348Z
paper_authors._id: 638ee2ed848625f9fad11034
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670309301345-638ee2ed848625f9fad11034.jpeg
paper_authors.fullname: Besmira Nushi
paper_authors.type: user
paper_authors.user: nushib
paper_authors._id: 6621ced43ec06d6503eecddb
paper_authors.name: Luis Oala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:03:00.478Z
paper_authors._id: 61170637468eb5fb52c297cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1640126895078-61170637468eb5fb52c297cd.jpeg
paper_authors.fullname: Luis Oala
paper_authors.type: user
paper_authors.user: luisoala
paper_authors._id: 6621ced43ec06d6503eecddc
paper_authors.name: Iftach Orr
paper_authors._id: 6621ced43ec06d6503eecddd
paper_authors.name: Alicia Parrish
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:03:13.916Z
paper_authors._id: 659d71eaf0ae1a57ae140831
paper_authors.avatarUrl: /avatars/b9eb8c70b4fe5dbd0f1555602ae41b65.svg
paper_authors.fullname: Alicia Parrish
paper_authors.type: user
paper_authors.user: avparrish
paper_authors._id: 6621ced43ec06d6503eecdde
paper_authors.name: Cigdem Patlak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:03:31.186Z
paper_authors._id: 65ba06bbb4ff0b04a992190d
paper_authors.avatarUrl: /avatars/98ddfef042298671705289d15247bfbd.svg
paper_authors.fullname: Cigdem Patlak
paper_authors.type: user
paper_authors.user: Cigdemp
paper_authors._id: 6621ced43ec06d6503eecddf
paper_authors.name: William Pietri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:03:23.709Z
paper_authors._id: 5f19c119925b9863e28ad645
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1595523321955-noauth.png
paper_authors.fullname: William Pietri
paper_authors.type: user
paper_authors.user: wpietri
paper_authors._id: 6621ced43ec06d6503eecde0
paper_authors.name: Forough Poursabzi-Sangdeh
paper_authors._id: 6621ced43ec06d6503eecde1
paper_authors.name: Eleonora Presani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:02:09.187Z
paper_authors._id: 64b19e3d93679cd00911cdd1
paper_authors.avatarUrl: /avatars/bab5a4543ce4f657415f7f36ca4fc6fb.svg
paper_authors.fullname: Eleonora Presani
paper_authors.type: user
paper_authors.user: epresani
paper_authors._id: 6621ced43ec06d6503eecde2
paper_authors.name: Fabrizio Puletti
paper_authors._id: 6621ced43ec06d6503eecde3
paper_authors.name: Paul Röttger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:02:00.949Z
paper_authors._id: 602ce925374a0dbe5856eca1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/602ce925374a0dbe5856eca1/lBvTn6hYOCF0ggD-rk_mf.jpeg
paper_authors.fullname: Paul Röttger
paper_authors.type: user
paper_authors.user: Paul
paper_authors._id: 6621ced43ec06d6503eecde4
paper_authors.name: Saurav Sahay
paper_authors._id: 6621ced43ec06d6503eecde5
paper_authors.name: Tim Santos
paper_authors._id: 6621ced43ec06d6503eecde6
paper_authors.name: Nino Scherrer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:01:43.629Z
paper_authors._id: 643dc14d37ca9134b589bfe0
paper_authors.avatarUrl: /avatars/998977d708f205dd5df430705e55ebc9.svg
paper_authors.fullname: Nino Scherrer
paper_authors.type: user
paper_authors.user: ninoscherrer
paper_authors._id: 6621ced43ec06d6503eecde7
paper_authors.name: Alice Schoenauer Sebag
paper_authors._id: 6621ced43ec06d6503eecde8
paper_authors.name: Patrick Schramowski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:01:25.795Z
paper_authors._id: 62d021a3dd7bdfc5e5c61c5c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d021a3dd7bdfc5e5c61c5c/bnQW2SqirfGaQmI84HW_c.jpeg
paper_authors.fullname: Patrick Schramowski
paper_authors.type: user
paper_authors.user: PSaiml
paper_authors._id: 6621ced43ec06d6503eecde9
paper_authors.name: Abolfazl Shahbazi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:01:16.527Z
paper_authors._id: 632a432774bc199012a0c84b
paper_authors.avatarUrl: /avatars/67d988b255c5ff37b23d595c26828cb4.svg
paper_authors.fullname: Abolfazl Shahbazi
paper_authors.type: user
paper_authors.user: ashahba
paper_authors._id: 6621ced43ec06d6503eecdea
paper_authors.name: Vin Sharma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:01:08.609Z
paper_authors._id: 651dbab62c6fac9717218917
paper_authors.avatarUrl: /avatars/5d8eeacb4d37e716192ff693667cb328.svg
paper_authors.fullname: Vin Sharma
paper_authors.type: user
paper_authors.user: ciphertext
paper_authors._id: 6621ced43ec06d6503eecdeb
paper_authors.name: Xudong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:01:01.731Z
paper_authors._id: 63c628610be43255f36f2a40
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1673930799006-noauth.jpeg
paper_authors.fullname: Xudong Shen
paper_authors.type: user
paper_authors.user: XudongShen
paper_authors._id: 6621ced43ec06d6503eecdec
paper_authors.name: Vamsi Sistla
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:00:53.803Z
paper_authors._id: 640a093996aae6497413e052
paper_authors.avatarUrl: /avatars/d9a979c1cd83fb8c7cbfdec49c67f88c.svg
paper_authors.fullname: Vamsi Sistla
paper_authors.type: user
paper_authors.user: Vsistla
paper_authors._id: 6621ced43ec06d6503eecded
paper_authors.name: Leonard Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:00:45.603Z
paper_authors._id: 64c13ee9e98a5e02c93459ee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c13ee9e98a5e02c93459ee/o7huQQPXZ5vS8r5RVDvWP.png
paper_authors.fullname: Leonard Tang
paper_authors.type: user
paper_authors.user: leonardtang
paper_authors._id: 6621ced43ec06d6503eecdee
paper_authors.name: Davide Testuggine
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:00:34.663Z
paper_authors._id: 64e927131d8b651a9ac0bee4
paper_authors.avatarUrl: /avatars/c74a3d2b8ffd75afcad35da39f5fad51.svg
paper_authors.fullname: Davide Testuggine
paper_authors.type: user
paper_authors.user: darktex
paper_authors._id: 6621ced43ec06d6503eecdef
paper_authors.name: Vithursan Thangarasa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:00:24.483Z
paper_authors._id: 64236f9523b98d5ee6ea2f61
paper_authors.avatarUrl: /avatars/e7f8156637aa8b0d36ce1954eac9dd77.svg
paper_authors.fullname: Vithursan Thangarasa
paper_authors.type: user
paper_authors.user: vithursant
paper_authors._id: 6621ced43ec06d6503eecdf0
paper_authors.name: Elizabeth Anne Watkins
paper_authors._id: 6621ced43ec06d6503eecdf1
paper_authors.name: Rebecca Weiss
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T09:00:07.191Z
paper_authors._id: 6430688e0d7e3248d0616a64
paper_authors.avatarUrl: /avatars/6d3ff97af3dd0da6f4781523e8cb2778.svg
paper_authors.fullname: Rebecca Weiss
paper_authors.type: user
paper_authors.user: rjweiss
paper_authors._id: 6621ced43ec06d6503eecdf2
paper_authors.name: Chris Welty
paper_authors._id: 6621ced43ec06d6503eecdf3
paper_authors.name: Tyler Wilbers
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:59:54.125Z
paper_authors._id: 648bd280b1793ce134986343
paper_authors.avatarUrl: /avatars/4c44a1b87e3a421a628a01956498a30d.svg
paper_authors.fullname: Tyler Wilbers
paper_authors.type: user
paper_authors.user: tybrs
paper_authors._id: 6621ced43ec06d6503eecdf4
paper_authors.name: Adina Williams
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:59:46.979Z
paper_authors._id: 6487618920fb2e386b73e5d2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6487618920fb2e386b73e5d2/ZnmBCV-gL3eMeDGC38Rz8.jpeg
paper_authors.fullname: Adina Williams
paper_authors.type: user
paper_authors.user: adinawilliams2
paper_authors._id: 6621ced43ec06d6503eecdf5
paper_authors.name: Carole-Jean Wu
paper_authors._id: 6621ced43ec06d6503eecdf6
paper_authors.name: Poonam Yadav
paper_authors._id: 6621ced43ec06d6503eecdf7
paper_authors.name: Xianjun Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-19T07:38:33.285Z
paper_authors._id: 604137e5c15e823a685f2af2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/604137e5c15e823a685f2af2/rrxZUyrbLpKqCmG6xtP8v.jpeg
paper_authors.fullname: Xianjun Yang
paper_authors.type: user
paper_authors.user: Xianjun
paper_authors._id: 6621ced43ec06d6503eecdf8
paper_authors.name: Yi Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:59:30.689Z
paper_authors._id: 63baf3da9726f7e58f8da2de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1673196486033-noauth.jpeg
paper_authors.fullname: Yi Zeng
paper_authors.type: user
paper_authors.user: yizeng
paper_authors._id: 6621ced43ec06d6503eecdf9
paper_authors.name: Wenhui Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:58:46.371Z
paper_authors._id: 658973b761f2dd8f6698c149
paper_authors.avatarUrl: /avatars/a6225d23b4f6508e7a799c08efac5464.svg
paper_authors.fullname: Wenhui Zhang
paper_authors.type: user
paper_authors.user: Shenqibaobei
paper_authors._id: 6621ced43ec06d6503eecdfa
paper_authors.name: Fedor Zhdanov
paper_authors._id: 6621ced43ec06d6503eecdfb
paper_authors.name: Jiacheng Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:58:05.110Z
paper_authors._id: 653801b167325b6218ddfdc8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653801b167325b6218ddfdc8/y3DjEDpcvgnRC30201dFc.jpeg
paper_authors.fullname: Jiacheng Zhu
paper_authors.type: user
paper_authors.user: JiachengZhu
paper_authors._id: 6621ced43ec06d6503eecdfc
paper_authors.name: Percy Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:57:31.374Z
paper_authors._id: 6409651b9e9f790c905b2335
paper_authors.avatarUrl: /avatars/1fb8c80b60f21f65a0a027319101f236.svg
paper_authors.fullname: Percy Liang
paper_authors.type: user
paper_authors.user: percyliang
paper_authors._id: 6621ced43ec06d6503eecdfd
paper_authors.name: Peter Mattson
paper_authors._id: 6621ced43ec06d6503eecdfe
paper_authors.name: Joaquin Vanschoren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:57:17.608Z
paper_authors._id: 6380c629e479bc19cbd4df0e
paper_authors.avatarUrl: /avatars/146df6eb6b1a94e7bdf8c679bfd171af.svg
paper_authors.fullname: Joaquin Vanschoren
paper_authors.type: user
paper_authors.user: JoaquinVanschoren
paper_title: Introducing v0.5 of the AI Safety Benchmark from MLCommons
paper_summary: This paper introduces v0.5 of the AI Safety Benchmark, which has been created
  by the MLCommons AI Safety Working Group. The AI Safety Benchmark has been
  designed to assess the safety risks of AI systems that use chat-tuned language
  models. We introduce a principled approach to specifying and constructing the
  benchmark, which for v0.5 covers only a single use case (an adult chatting to a
  general-purpose assistant in English), and a limited set of personas (i.e.,
  typical users, malicious users, and vulnerable users). We created a new
  taxonomy of 13 hazard categories, of which 7 have tests in the v0.5 benchmark.
  We plan to release version 1.0 of the AI Safety Benchmark by the end of 2024.
  The v1.0 benchmark will provide meaningful insights into the safety of AI
  systems. However, the v0.5 benchmark should not be used to assess the safety of
  AI systems. We have sought to fully document the limitations, flaws, and
  challenges of v0.5. This release of v0.5 of the AI Safety Benchmark includes
  (1) a principled approach to specifying and constructing the benchmark, which
  comprises use cases, types of systems under test (SUTs), language and context,
  personas, tests, and test items; (2) a taxonomy of 13 hazard categories with
  definitions and subcategories; (3) tests for seven of the hazard categories,
  each comprising a unique set of test items, i.e., prompts. There are 43,090
  test items in total, which we created with templates; (4) a grading system for
  AI systems against the benchmark; (5) an openly available platform, and
  downloadable tool, called ModelBench that can be used to evaluate the safety of
  AI systems on the benchmark; (6) an example evaluation report which benchmarks
  the performance of over a dozen openly available chat-tuned language models;
  (7) a test specification for the benchmark.
paper_ai_summary: The AI Safety Benchmark v0.5, developed by MLCommons, assesses the safety risks of chat-tuned language models using a taxonomy of 13 hazard categories and includes 43,090 test items.
paper_ai_keywords: chat-tuned language models hazard categories test items ModelBench evaluation report