From traces to measures: Large language models as a tool for psychological measurement from text
<|reference_start|>From traces to measures: Large language models as a tool for psychological measurement from text: Digital trace data provide potentially valuable resources for understanding human behaviour, but their value has been limited by issues of unclear measurement. The growth of large language models provides an opportunity to address this limitation in the case of text data. Specifically, recognizing cases where their responses are a form of psychological measurement (the use of observable indicators to assess an underlying construct) allows existing measures and accuracy assessment frameworks from psychology to be re-purposed to use with large language models. Based on this, we offer four methodological recommendations for using these models to quantify text features: (1) identify the target of measurement, (2) use multiple prompts, (3) assess internal consistency, and (4) treat evaluation metrics (such as human annotations) as expected correlates rather than direct ground-truth measures. Additionally, we provide a workflow for implementing this approach.<|reference_end|>
@article{simons2024from,
    title={From traces to measures: Large language models as a tool for
  psychological measurement from text},
    author={Joseph J.P. Simons, Wong Liang Ze, Prasanta Bhattacharya, Brandon
  Siyuan Loh, Wei Gao},
    journal={arXiv preprint arXiv:2405.07447},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.07447},
    primaryClass={cs.HC}
}