title: EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental
    Health Safety
summary: The rise of LLM-driven AI characters raises safety concerns, particularly for
  vulnerable human users with psychological disorders. To address these risks, we
  propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate
  mental health hazards in human-AI interactions. EmoAgent comprises two
  components: EmoEval simulates virtual users, including those portraying
  mentally vulnerable individuals, to assess mental health changes before and
  after interactions with AI characters. It uses clinically proven psychological
  and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks
  induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental
  status, predicting potential harm, and providing corrective feedback to
  mitigate risks. Experiments conducted in popular character-based chatbots show
  that emotionally engaging dialogues can lead to psychological deterioration in
  vulnerable users, with mental state deterioration in more than 34.4% of the
  simulations. EmoGuard significantly reduces these deterioration rates,
  underscoring its role in ensuring safer AI-human interactions. Our code is
  available at: https://github.com/1akaman/EmoAgent
paper_authors._id: 67fdc937089aec0f3b154dd7
paper_authors.name: Jiahao Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T19:14:59.480Z
paper_authors._id: 65271df5abd7795aaa1fd86e
paper_authors.avatarUrl: /avatars/99e1aabaefcf5e7f438817375cd1ddef.svg
paper_authors.fullname: Jiahao Qiu
paper_authors.type: user
paper_authors.user: jiahaoq
paper_authors._id: 67fdc937089aec0f3b154dd8
paper_authors.name: Yinghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:30:49.206Z
paper_authors._id: 652abf5360e706730596e8f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/zRFJ4FjZJZGkz2wH8PPmU.jpeg
paper_authors.fullname: Yinghui He
paper_authors.type: user
paper_authors.user: yinghuihe
paper_authors._id: 67fdc937089aec0f3b154dd9
paper_authors.name: Xinzhe Juan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:30:56.417Z
paper_authors._id: 674500b57a76d46e9141af8b
paper_authors.avatarUrl: /avatars/e70243c31e2ac9f000542e8504e65b51.svg
paper_authors.fullname: Xinzhe Juan
paper_authors.type: user
paper_authors.user: ChrisJuan
paper_authors._id: 67fdc937089aec0f3b154dda
paper_authors.name: Yiming Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T09:42:50.544Z
paper_authors._id: 67a5ae48166721c8f99f8dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/QogTIecNITyOGrsqkJzWK.png
paper_authors.fullname: Yimin Wang
paper_authors.type: user
paper_authors.user: 99sweetcookie
paper_authors._id: 67fdc937089aec0f3b154ddb
paper_authors.name: Yuhan Liu
paper_authors._id: 67fdc937089aec0f3b154ddc
paper_authors.name: Zixin Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:31:38.642Z
paper_authors._id: 6657f2041d83f3ee61bf414d
paper_authors.avatarUrl: /avatars/e678fbbba2e93536dfc702e8ae629a95.svg
paper_authors.fullname: zixin
paper_authors.type: user
paper_authors.user: yaozixin
paper_authors._id: 67fdc937089aec0f3b154ddd
paper_authors.name: Yue Wu
paper_authors._id: 67fdc937089aec0f3b154dde
paper_authors.name: Xun Jiang
paper_authors._id: 67fdc937089aec0f3b154ddf
paper_authors.name: Ling Yang
paper_authors._id: 67fdc937089aec0f3b154de0
paper_authors.name: Mengdi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:32:09.308Z
paper_authors._id: 6599415e8c8ac79295e0b5e3
paper_authors.avatarUrl: /avatars/85500bc8d2cd51444adcc19b1f8db313.svg
paper_authors.fullname: Mengdi Wang
paper_authors.type: user
paper_authors.user: Edify-Kd2024
paper_title: EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental
    Health Safety
paper_summary: The rise of LLM-driven AI characters raises safety concerns, particularly for
  vulnerable human users with psychological disorders. To address these risks, we
  propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate
  mental health hazards in human-AI interactions. EmoAgent comprises two
  components: EmoEval simulates virtual users, including those portraying
  mentally vulnerable individuals, to assess mental health changes before and
  after interactions with AI characters. It uses clinically proven psychological
  and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks
  induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental
  status, predicting potential harm, and providing corrective feedback to
  mitigate risks. Experiments conducted in popular character-based chatbots show
  that emotionally engaging dialogues can lead to psychological deterioration in
  vulnerable users, with mental state deterioration in more than 34.4% of the
  simulations. EmoGuard significantly reduces these deterioration rates,
  underscoring its role in ensuring safer AI-human interactions. Our code is
  available at: https://github.com/1akaman/EmoAgent
paper_ai_summary: EmoAgent, a multi-agent AI framework, evaluates and mitigates mental health hazards by simulating user interactions and providing corrective feedback.
paper_ai_keywords: LLM EmoAgent EmoEval EmoGuard PHQ-9 PDI PANSS virtual users mentally vulnerable individuals mental health changes psychological deterioration AI-human interactions