Quantifying AI Psychology: A Psychometrics Benchmark for Large Language Models
<|reference_start|>Quantifying AI Psychology: A Psychometrics Benchmark for Large Language Models: Large Language Models (LLMs) have demonstrated exceptional task-solving capabilities, increasingly adopting roles akin to human-like assistants. The broader integration of LLMs into society has sparked interest in whether they manifest psychological attributes, and whether these attributes are stable-inquiries that could deepen the understanding of their behaviors. Inspired by psychometrics, this paper presents a framework for investigating psychology in LLMs, including psychological dimension identification, assessment dataset curation, and assessment with results validation. Following this framework, we introduce a comprehensive psychometrics benchmark for LLMs that covers six psychological dimensions: personality, values, emotion, theory of mind, motivation, and intelligence. This benchmark includes thirteen datasets featuring diverse scenarios and item types. Our findings indicate that LLMs manifest a broad spectrum of psychological attributes. We also uncover discrepancies between LLMs' self-reported traits and their behaviors in real-world scenarios. This paper demonstrates a thorough psychometric assessment of LLMs, providing insights into reliable evaluation and potential applications in AI and social sciences.<|reference_end|>
@article{li2024quantifying,
    title={Quantifying AI Psychology: A Psychometrics Benchmark for Large Language
  Models},
    author={Yuan Li, Yue Huang, Hongyi Wang, Xiangliang Zhang, James Zou, Lichao
  Sun},
    journal={arXiv preprint arXiv:2406.17675},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.17675},
    primaryClass={cs.CL}
}