title: CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from
    Multiple Perspectives
summary: Navigating high-stakes dilemmas involving conflicting values is challenging
  even for humans, let alone for AI. Yet prior work in evaluating the reasoning
  capabilities of large language models (LLMs) in such situations has been
  limited to everyday scenarios. To close this gap, this work first introduces
  CLASH (Character perspective-based LLM Assessments in Situations with
  High-stakes), a meticulously curated dataset consisting of 345 high-impact
  dilemmas along with 3,795 individual perspectives of diverse values. In
  particular, we design CLASH in a way to support the study of critical aspects
  of value-based decision-making processes which are missing from prior work,
  including understanding decision ambivalence and psychological discomfort as
  well as capturing the temporal shifts of values in characters' perspectives. By
  benchmarking 10 open and closed frontier models, we uncover several key
  findings. (1) Even the strongest models, such as GPT-4o and Claude-Sonnet,
  achieve less than 50% accuracy in identifying situations where the decision
  should be ambivalent, while they perform significantly better in clear-cut
  scenarios. (2) While LLMs reasonably predict psychological discomfort as marked
  by human, they inadequately comprehend perspectives involving value shifts,
  indicating a need for LLMs to reason over complex values. (3) Our experiments
  also reveal a significant correlation between LLMs' value preferences and their
  steerability towards a given value. (4) Finally, LLMs exhibit greater
  steerability when engaged in value reasoning from a third-party perspective,
  compared to a first-person setup, though certain value pairs benefit uniquely
  from the first-person framing.
paper_authors._id: 68003da4c771c307fec04da7
paper_authors.name: Ayoung Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:03:37.028Z
paper_authors._id: 66d079d94b3b38cefaf1dc4e
paper_authors.avatarUrl: /avatars/f680a19fb7b8e52c811eb6df218a2cea.svg
paper_authors.fullname: Lee
paper_authors.type: user
paper_authors.user: Ayoung01
paper_authors._id: 68003da4c771c307fec04da8
paper_authors.name: Ryan Sungmo Kwon
paper_authors._id: 68003da4c771c307fec04da9
paper_authors.name: Peter Railton
paper_authors._id: 68003da4c771c307fec04daa
paper_authors.name: Lu Wang
paper_title: CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from
    Multiple Perspectives
paper_summary: Navigating high-stakes dilemmas involving conflicting values is challenging
  even for humans, let alone for AI. Yet prior work in evaluating the reasoning
  capabilities of large language models (LLMs) in such situations has been
  limited to everyday scenarios. To close this gap, this work first introduces
  CLASH (Character perspective-based LLM Assessments in Situations with
  High-stakes), a meticulously curated dataset consisting of 345 high-impact
  dilemmas along with 3,795 individual perspectives of diverse values. In
  particular, we design CLASH in a way to support the study of critical aspects
  of value-based decision-making processes which are missing from prior work,
  including understanding decision ambivalence and psychological discomfort as
  well as capturing the temporal shifts of values in characters' perspectives. By
  benchmarking 10 open and closed frontier models, we uncover several key
  findings. (1) Even the strongest models, such as GPT-4o and Claude-Sonnet,
  achieve less than 50% accuracy in identifying situations where the decision
  should be ambivalent, while they perform significantly better in clear-cut
  scenarios. (2) While LLMs reasonably predict psychological discomfort as marked
  by human, they inadequately comprehend perspectives involving value shifts,
  indicating a need for LLMs to reason over complex values. (3) Our experiments
  also reveal a significant correlation between LLMs' value preferences and their
  steerability towards a given value. (4) Finally, LLMs exhibit greater
  steerability when engaged in value reasoning from a third-party perspective,
  compared to a first-person setup, though certain value pairs benefit uniquely
  from the first-person framing.
paper_ai_summary: LLMs perform well in clear-cut ethical dilemmas but struggle with ambiguity, psychological discomfort, and value shifts in high-stakes scenarios, showing differences in accuracy and steerability based on perspective.
paper_ai_keywords: large language models LLMs CLASH high-impact dilemmas decision ambivalence psychological discomfort value shifts value preferences steerability third-party perspective first-person perspective