Across reliability results, reported in Section 5.2.2, internal consistency reliability (α and
λ6) improve after instruction fine-tuning. However, factor saturation (captured in McDonald’s
ω) does not improve; it is indistinguishably high for both base and instruction fine-tuned
models of the same size (PaLM, Flan-PaLM, and Flan-PaLMChilla). How is it possible
that base model’s (PaLM 62B) responses are unidimensional (e.g., their variance reflects
one underlying factor in statistical analysis) but not internally consistent (e.g., generally

31

coherent when measured multiple times)? We turn to a possible explanation from human
psychometrics. Humans can generate unidimensional responses to questionnaires that are
simultaneously inconsistent when the questionnaire items 1) have varying levels of difficulty
or 2) are actually measuring different underlying psychological constructs.
PersonalityTraitsinLargeLanguageModels