Measuring Human and AI Values based on Generative Psychometrics with
  Large Language Models
Human values and their measurement are long-standing interdisciplinary
inquiry. Recent advances in AI have sparked renewed interest in this area, with
large language models (LLMs) emerging as both tools and subjects of value
measurement. This work introduces Generative Psychometrics for Values (GPV), an
LLM-based, data-driven value measurement paradigm, theoretically grounded in
text-revealed selective perceptions. We begin by fine-tuning an LLM for
accurate perception-level value measurement and verifying the capability of
LLMs to parse texts into perceptions, forming the core of the GPV pipeline.
Applying GPV to human-authored blogs, we demonstrate its stability, validity,
and superiority over prior psychological tools. Then, extending GPV to LLM
value measurement, we advance the current art with 1) a psychometric
methodology that measures LLM values based on their scalable and free-form
outputs, enabling context-specific measurement; 2) a comparative analysis of
measurement paradigms, indicating response biases of prior methods; and 3) an
attempt to bridge LLM values and their safety, revealing the predictive power
of different value systems and the impacts of various values on LLM safety.
Through interdisciplinary efforts, we aim to leverage AI for next-generation
psychometrics and psychometrics for value-aligned AI.
66eb9e6c052019916e420929
Haoran Ye
66eb9e6c052019916e42092a
Yuhang Xie
66eb9e6c052019916e42092b
Yuanyi Ren
66eb9e6c052019916e42092c
Hanjun Fang
66eb9e6c052019916e42092d
Xin Zhang
66eb9e6c052019916e42092e
Guojie Song
Measuring Human and AI Values based on Generative Psychometrics with
  Large Language Models
Human values and their measurement are long-standing interdisciplinary
inquiry. Recent advances in AI have sparked renewed interest in this area, with
large language models (LLMs) emerging as both tools and subjects of value
measurement. This work introduces Generative Psychometrics for Values (GPV), an
LLM-based, data-driven value measurement paradigm, theoretically grounded in
text-revealed selective perceptions. We begin by fine-tuning an LLM for
accurate perception-level value measurement and verifying the capability of
LLMs to parse texts into perceptions, forming the core of the GPV pipeline.
Applying GPV to human-authored blogs, we demonstrate its stability, validity,
and superiority over prior psychological tools. Then, extending GPV to LLM
value measurement, we advance the current art with 1) a psychometric
methodology that measures LLM values based on their scalable and free-form
outputs, enabling context-specific measurement; 2) a comparative analysis of
measurement paradigms, indicating response biases of prior methods; and 3) an
attempt to bridge LLM values and their safety, revealing the predictive power
of different value systems and the impacts of various values on LLM safety.
Through interdisciplinary efforts, we aim to leverage AI for next-generation
psychometrics and psychometrics for value-aligned AI.
Generative Psychometrics for Values (GPV) uses large language models to measure human and LLM values, demonstrating stability and superiority over prior methods while highlighting the connection between values and AI safety.
Generative Psychometrics for Values large language models LLM fine-tuning text-revealed selective perceptions psychometric methodology scalable and free-form outputs value measurement response biases AI safety