title: Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in
    Large Language Models
summary: Assessing how well a large language model (LLM) understands human, rather
  than merely text, remains an open challenge. To bridge the gap, we introduce
  Sentient Agent as a Judge (SAGE), an automated evaluation framework that
  measures an LLM's higher-order social cognition. SAGE instantiates a Sentient
  Agent that simulates human-like emotional changes and inner thoughts during
  interaction, providing a more realistic evaluation of the tested model in
  multi-turn conversations. At every turn, the agent reasons about (i) how its
  emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a
  numerical emotion trajectory and interpretable inner thoughts. Experiments on
  100 supportive-dialogue scenarios show that the final Sentient emotion score
  correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings
  and utterance-level empathy metrics, validating psychological fidelity. We also
  build a public Sentient Leaderboard covering 18 commercial and open-source
  models that uncovers substantial gaps (up to 4x) between frontier systems
  (GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in
  conventional leaderboards (e.g., Arena). SAGE thus provides a principled,
  scalable and interpretable tool for tracking progress toward genuinely
  empathetic and socially adept language agents.
paper_authors._id: 681d7031e9969eecfcb4eb81
paper_authors.name: Bang Zhang
paper_authors._id: 681d7031e9969eecfcb4eb82
paper_authors.name: Ruotian Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-09T07:21:30.886Z
paper_authors._id: 648294b2eb4befee378951c1
paper_authors.avatarUrl: /avatars/da5d8bf9d8662cc2ffa2c0de49bd66a3.svg
paper_authors.fullname: Ruotian Ma
paper_authors.type: user
paper_authors.user: vvibt
paper_authors._id: 681d7031e9969eecfcb4eb83
paper_authors.name: Qingxuan Jiang
paper_authors._id: 681d7031e9969eecfcb4eb84
paper_authors.name: Peisong Wang
paper_authors._id: 681d7031e9969eecfcb4eb85
paper_authors.name: Jiaqi Chen
paper_authors._id: 681d7031e9969eecfcb4eb86
paper_authors.name: Zheng Xie
paper_authors._id: 681d7031e9969eecfcb4eb87
paper_authors.name: Xingyu Chen
paper_authors._id: 681d7031e9969eecfcb4eb88
paper_authors.name: Yue Wang
paper_authors._id: 681d7031e9969eecfcb4eb89
paper_authors.name: Fanghua Ye
paper_authors._id: 681d7031e9969eecfcb4eb8a
paper_authors.name: Jian Li
paper_authors._id: 681d7031e9969eecfcb4eb8b
paper_authors.name: Yifan Yang
paper_authors._id: 681d7031e9969eecfcb4eb8c
paper_authors.name: Zhaopeng Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-09T07:21:28.468Z
paper_authors._id: 67485743561b1e6f9579389f
paper_authors.avatarUrl: /avatars/8a4cc63bd7be388010bc329bb74582a1.svg
paper_authors.fullname: Zhaopeng Tu
paper_authors.type: user
paper_authors.user: zptu
paper_authors._id: 681d7031e9969eecfcb4eb8d
paper_authors.name: Xiaolong Li
paper_title: Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in
    Large Language Models
paper_summary: Assessing how well a large language model (LLM) understands human, rather
  than merely text, remains an open challenge. To bridge the gap, we introduce
  Sentient Agent as a Judge (SAGE), an automated evaluation framework that
  measures an LLM's higher-order social cognition. SAGE instantiates a Sentient
  Agent that simulates human-like emotional changes and inner thoughts during
  interaction, providing a more realistic evaluation of the tested model in
  multi-turn conversations. At every turn, the agent reasons about (i) how its
  emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a
  numerical emotion trajectory and interpretable inner thoughts. Experiments on
  100 supportive-dialogue scenarios show that the final Sentient emotion score
  correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings
  and utterance-level empathy metrics, validating psychological fidelity. We also
  build a public Sentient Leaderboard covering 18 commercial and open-source
  models that uncovers substantial gaps (up to 4x) between frontier systems
  (GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in
  conventional leaderboards (e.g., Arena). SAGE thus provides a principled,
  scalable and interpretable tool for tracking progress toward genuinely
  empathetic and socially adept language agents.
paper_ai_summary: SAGE, an automated evaluation framework, assesses the higher-order social cognition and empathy of large language models through simulated human-like emotions and inner thoughts in multi-turn conversations.
paper_ai_keywords: Sentient Agent as a Judge SAGE higher-order social cognition emotional changes inner thoughts numerical emotion trajectory Barrett-Lennard Relationship Inventory empathy metrics Sentient Leaderboard empathetic language agents