Phare: A Safety Probe for Large Language Models
Ensuring the safety of large language models (LLMs) is critical for
responsible deployment, yet existing evaluations often prioritize performance
over identifying failure modes. We introduce Phare, a multilingual diagnostic
framework to probe and evaluate LLM behavior across three critical dimensions:
hallucination and reliability, social biases, and harmful content generation.
Our evaluation of 17 state-of-the-art LLMs reveals patterns of systematic
vulnerabilities across all safety dimensions, including sycophancy, prompt
sensitivity, and stereotype reproduction. By highlighting these specific
failure modes rather than simply ranking models, Phare provides researchers and
practitioners with actionable insights to build more robust, aligned, and
trustworthy language systems.
682d7ae082567fffe108b31a
Pierre Le Jeune
extracted_confirmed
2025-05-21T07:16:41.247Z
6596ca5cce76219628b8eab4
/avatars/51cdea4e1e0e53260d403ceb7bc6de90.svg
Pierre Le Jeune
user
pierlj
682d7ae082567fffe108b31b
Benoît Malézieux
extracted_pending
2025-05-21T07:04:02.669Z
65bccff4c1a44b6ef1c100da
/avatars/bf91000c78b83167958dc44c582397f0.svg
benoit
user
bmalezieux
682d7ae082567fffe108b31c
Weixuan Xiao
extracted_pending
2025-05-21T07:04:02.669Z
649f00fc37bfb5202be464a9
/avatars/89f6c6a92c076099f5450c3cd2057619.svg
Weixuan Xiao
user
weixuan-giskard
682d7ae082567fffe108b31d
Matteo Dora
Phare: A Safety Probe for Large Language Models
Ensuring the safety of large language models (LLMs) is critical for
responsible deployment, yet existing evaluations often prioritize performance
over identifying failure modes. We introduce Phare, a multilingual diagnostic
framework to probe and evaluate LLM behavior across three critical dimensions:
hallucination and reliability, social biases, and harmful content generation.
Our evaluation of 17 state-of-the-art LLMs reveals patterns of systematic
vulnerabilities across all safety dimensions, including sycophancy, prompt
sensitivity, and stereotype reproduction. By highlighting these specific
failure modes rather than simply ranking models, Phare provides researchers and
practitioners with actionable insights to build more robust, aligned, and
trustworthy language systems.
Phare evaluates large language models across safety dimensions to uncover specific failure modes, offering insights for building more robust systems.
large language models hallucination reliability social biases harmful content generation sycophancy prompt sensitivity stereotype reproduction