EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental
  Health Safety
The rise of LLM-driven AI characters raises safety concerns, particularly for
vulnerable human users with psychological disorders. To address these risks, we
propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate
mental health hazards in human-AI interactions. EmoAgent comprises two
components: EmoEval simulates virtual users, including those portraying
mentally vulnerable individuals, to assess mental health changes before and
after interactions with AI characters. It uses clinically proven psychological
and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks
induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental
status, predicting potential harm, and providing corrective feedback to
mitigate risks. Experiments conducted in popular character-based chatbots show
that emotionally engaging dialogues can lead to psychological deterioration in
vulnerable users, with mental state deterioration in more than 34.4% of the
simulations. EmoGuard significantly reduces these deterioration rates,
underscoring its role in ensuring safer AI-human interactions. Our code is
available at: https://github.com/1akaman/EmoAgent
67fdc937089aec0f3b154dd7
Jiahao Qiu
claimed_verified
2025-05-28T19:14:59.480Z
65271df5abd7795aaa1fd86e
/avatars/99e1aabaefcf5e7f438817375cd1ddef.svg
Jiahao Qiu
user
jiahaoq
67fdc937089aec0f3b154dd8
Yinghui He
admin_assigned
2025-04-15T08:30:49.206Z
652abf5360e706730596e8f4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/zRFJ4FjZJZGkz2wH8PPmU.jpeg
Yinghui He
user
yinghuihe
67fdc937089aec0f3b154dd9
Xinzhe Juan
admin_assigned
2025-04-15T08:30:56.417Z
674500b57a76d46e9141af8b
/avatars/e70243c31e2ac9f000542e8504e65b51.svg
Xinzhe Juan
user
ChrisJuan
67fdc937089aec0f3b154dda
Yiming Wang
claimed_verified
2025-05-19T09:42:50.544Z
67a5ae48166721c8f99f8dac
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/QogTIecNITyOGrsqkJzWK.png
Yimin Wang
user
99sweetcookie
67fdc937089aec0f3b154ddb
Yuhan Liu
67fdc937089aec0f3b154ddc
Zixin Yao
admin_assigned
2025-04-15T08:31:38.642Z
6657f2041d83f3ee61bf414d
/avatars/e678fbbba2e93536dfc702e8ae629a95.svg
zixin
user
yaozixin
67fdc937089aec0f3b154ddd
Yue Wu
67fdc937089aec0f3b154dde
Xun Jiang
67fdc937089aec0f3b154ddf
Ling Yang
claimed_verified
2025-06-27T08:58:21.680Z
64fde4e252e82dd432b74ce9
https://cdn-avatars.huggingface.co/v1/production/uploads/64fde4e252e82dd432b74ce9/-CQZbBP7FsPPyawYrsi4z.jpeg
Ling Yang
user
Lingaaaaaaa
67fdc937089aec0f3b154de0
Mengdi Wang
admin_assigned
2025-04-15T08:32:09.308Z
6599415e8c8ac79295e0b5e3
/avatars/85500bc8d2cd51444adcc19b1f8db313.svg
Mengdi Wang
user
Edify-Kd2024
EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental
  Health Safety
The rise of LLM-driven AI characters raises safety concerns, particularly for
vulnerable human users with psychological disorders. To address these risks, we
propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate
mental health hazards in human-AI interactions. EmoAgent comprises two
components: EmoEval simulates virtual users, including those portraying
mentally vulnerable individuals, to assess mental health changes before and
after interactions with AI characters. It uses clinically proven psychological
and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks
induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental
status, predicting potential harm, and providing corrective feedback to
mitigate risks. Experiments conducted in popular character-based chatbots show
that emotionally engaging dialogues can lead to psychological deterioration in
vulnerable users, with mental state deterioration in more than 34.4% of the
simulations. EmoGuard significantly reduces these deterioration rates,
underscoring its role in ensuring safer AI-human interactions. Our code is
available at: https://github.com/1akaman/EmoAgent
EmoAgent, a multi-agent AI framework, evaluates and mitigates mental health hazards by simulating user interactions and providing corrective feedback.
LLM EmoAgent EmoEval EmoGuard PHQ-9 PDI PANSS virtual users mentally vulnerable individuals mental health changes psychological deterioration AI-human interactions