personality disorders [44]). Indeed, administering questionnaires in this way to LLMs
creates an arbitrary viewpoint from which to elicit human traits, and is likely biased
by the ordering of the questionnaire itself [45] and prompting the LLM to respond
in an interview setting (where it may respond differently knowing an interviewer is
observing). Each LLM response to a given questionnaire item was not an indepen-
dent event, but considered all previous responses shown in the transcript. Second, the
LLMs in these studies were not used deterministically. This not only hampers repro-
ducibility, but also poses implications for reliability. Computing reliability metrics for
questionnaires scored in this unconventional way is precarious because such reliability
metrics rely on item-level variance. If this item-level variance is contaminated by varia-
tion introduced by the model parameters in a different way for each item, it is difficult
PersonalityTraitsinLargeLanguageModels