title: WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language
    Models
summary: WalledEval is a comprehensive AI safety testing toolkit designed to evaluate
  large language models (LLMs). It accommodates a diverse range of models,
  including both open-weight and API-based ones, and features over 35 safety
  benchmarks covering areas such as multilingual safety, exaggerated safety, and
  prompt injections. The framework supports both LLM and judge benchmarking, and
  incorporates custom mutators to test safety against various text-style
  mutations such as future tense and paraphrasing. Additionally, WalledEval
  introduces WalledGuard, a new, small and performant content moderation tool,
  and SGXSTest, a benchmark for assessing exaggerated safety in cultural
  contexts. We make WalledEval publicly available at
  https://github.com/walledai/walledevalA.
paper_authors._id: 66b489ffd84a01d665127b6e
paper_authors.name: Prannaya Gupta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:34:42.340Z
paper_authors._id: 60a546bdf9b53404e7806278
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1621444268349-noauth.png
paper_authors.fullname: Prannaya Gupta
paper_authors.type: user
paper_authors.user: ThePyProgrammer
paper_authors._id: 66b489ffd84a01d665127b6f
paper_authors.name: Le Qi Yau
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:48:18.061Z
paper_authors._id: 64b35a5d4c3cc95a7553c360
paper_authors.avatarUrl: /avatars/1d96d3fcf42c141d5fb9ef7bd691eb96.svg
paper_authors.fullname: Yau Le Qi
paper_authors.type: user
paper_authors.user: teraSurfer40141
paper_authors._id: 66b489ffd84a01d665127b70
paper_authors.name: Hao Han Low
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:48:13.819Z
paper_authors._id: 669745a9c1d9adc0c7daf1f5
paper_authors.avatarUrl: /avatars/dac56ed2f6776074548b0132d097f13c.svg
paper_authors.fullname: Low Hao Han
paper_authors.type: user
paper_authors.user: redherring1
paper_authors._id: 66b489ffd84a01d665127b71
paper_authors.name: I-Shiang Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:48:16.221Z
paper_authors._id: 668ff9e81e3f5b16cef10003
paper_authors.avatarUrl: /avatars/4e10b275131058f588e228d09e5e7d24.svg
paper_authors.fullname: Lee I-Shiang
paper_authors.type: user
paper_authors.user: i-shiang
paper_authors._id: 66b489ffd84a01d665127b72
paper_authors.name: Hugo Maximus Lim
paper_authors._id: 66b489ffd84a01d665127b73
paper_authors.name: Yu Xin Teoh
paper_authors._id: 66b489ffd84a01d665127b74
paper_authors.name: Jia Hng Koh
paper_authors._id: 66b489ffd84a01d665127b75
paper_authors.name: Dar Win Liew
paper_authors._id: 66b489ffd84a01d665127b76
paper_authors.name: Rishabh Bhardwaj
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:35:27.248Z
paper_authors._id: 5f278507e923d665e616271b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f278507e923d665e616271b/tWFuswXOTXtvMdL8zSrr_.png
paper_authors.fullname: Rishabh Bhardwaj
paper_authors.type: user
paper_authors.user: RishabhBhardwaj
paper_authors._id: 66b489ffd84a01d665127b77
paper_authors.name: Rajat Bhardwaj
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:35:34.653Z
paper_authors._id: 6684cde7b6201f5294bdf41a
paper_authors.avatarUrl: /avatars/3bc9e27a102f7a1a503b70b43e9e7034.svg
paper_authors.fullname: Rajat Bhardwaj
paper_authors.type: user
paper_authors.user: bhardwajrajat81
paper_authors._id: 66b489ffd84a01d665127b78
paper_authors.name: Soujanya Poria
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:35:41.203Z
paper_authors._id: 626b626405fe1cb65725aca1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/aa-Lata46I3fXOmMetvXH.jpeg
paper_authors.fullname: Soujanya Poria
paper_authors.type: user
paper_authors.user: soujanyaporia
paper_title: WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language
    Models
paper_summary: WalledEval is a comprehensive AI safety testing toolkit designed to evaluate
  large language models (LLMs). It accommodates a diverse range of models,
  including both open-weight and API-based ones, and features over 35 safety
  benchmarks covering areas such as multilingual safety, exaggerated safety, and
  prompt injections. The framework supports both LLM and judge benchmarking, and
  incorporates custom mutators to test safety against various text-style
  mutations such as future tense and paraphrasing. Additionally, WalledEval
  introduces WalledGuard, a new, small and performant content moderation tool,
  and SGXSTest, a benchmark for assessing exaggerated safety in cultural
  contexts. We make WalledEval publicly available at
  https://github.com/walledai/walledevalA.
paper_ai_summary: WalledEval is a comprehensive safety testing toolkit for large language models using multiple safety benchmarks and custom mutators to assess performance in areas such as multilingual safety, exaggerated safety, and cultural contexts.
paper_ai_keywords: large language models safety benchmarks multilingual safety exaggerated safety prompt injections LLM judge benchmarking custom mutators WalledGuard SGXSTest