RealHarm: A Collection of Real-World Language Model Application Failures
Language model deployments in consumer-facing applications introduce numerous
risks. While existing research on harms and hazards of such applications
follows top-down approaches derived from regulatory frameworks and theoretical
analyses, empirical evidence of real-world failure modes remains underexplored.
In this work, we introduce RealHarm, a dataset of annotated problematic
interactions with AI agents built from a systematic review of publicly reported
incidents. Analyzing harms, causes, and hazards specifically from the
deployer's perspective, we find that reputational damage constitutes the
predominant organizational harm, while misinformation emerges as the most
common hazard category. We empirically evaluate state-of-the-art guardrails and
content moderation systems to probe whether such systems would have prevented
the incidents, revealing a significant gap in the protection of AI
applications.
67ff741e01428cf3b865ceb5
Pierre Le Jeune
claimed_verified
2025-04-16T09:15:33.830Z
6596ca5cce76219628b8eab4
/avatars/51cdea4e1e0e53260d403ceb7bc6de90.svg
Pierre Le Jeune
user
pierlj
67ff741e01428cf3b865ceb6
Jiaen Liu
67ff741e01428cf3b865ceb7
Luca Rossi
67ff741e01428cf3b865ceb8
Matteo Dora
RealHarm: A Collection of Real-World Language Model Application Failures
Language model deployments in consumer-facing applications introduce numerous
risks. While existing research on harms and hazards of such applications
follows top-down approaches derived from regulatory frameworks and theoretical
analyses, empirical evidence of real-world failure modes remains underexplored.
In this work, we introduce RealHarm, a dataset of annotated problematic
interactions with AI agents built from a systematic review of publicly reported
incidents. Analyzing harms, causes, and hazards specifically from the
deployer's perspective, we find that reputational damage constitutes the
predominant organizational harm, while misinformation emerges as the most
common hazard category. We empirically evaluate state-of-the-art guardrails and
content moderation systems to probe whether such systems would have prevented
the incidents, revealing a significant gap in the protection of AI
applications.
RealHarm, a dataset of real-world AI failures, reveals reputational damage and misinformation as primary risks in language model deployments, highlighting vulnerabilities in existing guardrails and content moderation systems.
Language model deployments AI agents RealHarm annotated problematic interactions systematic review empirical evidence organizational harm reputational damage hazard categories misinformation state-of-the-art guardrails content moderation systems