title: Measuring Human and AI Values based on Generative Psychometrics with
    Large Language Models
summary: Human values and their measurement are long-standing interdisciplinary
  inquiry. Recent advances in AI have sparked renewed interest in this area, with
  large language models (LLMs) emerging as both tools and subjects of value
  measurement. This work introduces Generative Psychometrics for Values (GPV), an
  LLM-based, data-driven value measurement paradigm, theoretically grounded in
  text-revealed selective perceptions. We begin by fine-tuning an LLM for
  accurate perception-level value measurement and verifying the capability of
  LLMs to parse texts into perceptions, forming the core of the GPV pipeline.
  Applying GPV to human-authored blogs, we demonstrate its stability, validity,
  and superiority over prior psychological tools. Then, extending GPV to LLM
  value measurement, we advance the current art with 1) a psychometric
  methodology that measures LLM values based on their scalable and free-form
  outputs, enabling context-specific measurement; 2) a comparative analysis of
  measurement paradigms, indicating response biases of prior methods; and 3) an
  attempt to bridge LLM values and their safety, revealing the predictive power
  of different value systems and the impacts of various values on LLM safety.
  Through interdisciplinary efforts, we aim to leverage AI for next-generation
  psychometrics and psychometrics for value-aligned AI.
paper_authors._id: 66eb9e6c052019916e420929
paper_authors.name: Haoran Ye
paper_authors._id: 66eb9e6c052019916e42092a
paper_authors.name: Yuhang Xie
paper_authors._id: 66eb9e6c052019916e42092b
paper_authors.name: Yuanyi Ren
paper_authors._id: 66eb9e6c052019916e42092c
paper_authors.name: Hanjun Fang
paper_authors._id: 66eb9e6c052019916e42092d
paper_authors.name: Xin Zhang
paper_authors._id: 66eb9e6c052019916e42092e
paper_authors.name: Guojie Song
paper_title: Measuring Human and AI Values based on Generative Psychometrics with
    Large Language Models
paper_summary: Human values and their measurement are long-standing interdisciplinary
  inquiry. Recent advances in AI have sparked renewed interest in this area, with
  large language models (LLMs) emerging as both tools and subjects of value
  measurement. This work introduces Generative Psychometrics for Values (GPV), an
  LLM-based, data-driven value measurement paradigm, theoretically grounded in
  text-revealed selective perceptions. We begin by fine-tuning an LLM for
  accurate perception-level value measurement and verifying the capability of
  LLMs to parse texts into perceptions, forming the core of the GPV pipeline.
  Applying GPV to human-authored blogs, we demonstrate its stability, validity,
  and superiority over prior psychological tools. Then, extending GPV to LLM
  value measurement, we advance the current art with 1) a psychometric
  methodology that measures LLM values based on their scalable and free-form
  outputs, enabling context-specific measurement; 2) a comparative analysis of
  measurement paradigms, indicating response biases of prior methods; and 3) an
  attempt to bridge LLM values and their safety, revealing the predictive power
  of different value systems and the impacts of various values on LLM safety.
  Through interdisciplinary efforts, we aim to leverage AI for next-generation
  psychometrics and psychometrics for value-aligned AI.
paper_ai_summary: Generative Psychometrics for Values (GPV) uses large language models to measure human and LLM values, demonstrating stability and superiority over prior methods while highlighting the connection between values and AI safety.
paper_ai_keywords: Generative Psychometrics for Values large language models LLM fine-tuning text-revealed selective perceptions psychometric methodology scalable and free-form outputs value measurement response biases AI safety