  Theory of mind (ToM) evaluations currently focus on testing models using
passive narratives that inherently lack interactivity. We introduce FANToM, a
new benchmark designed to stress-test ToM within information-asymmetric
conversational contexts via question answering. Our benchmark draws upon
important theoretical requisites from psychology and necessary empirical
considerations when evaluating large language models (LLMs). In particular, we
formulate multiple types of questions that demand the same underlying reasoning
to identify illusory or false sense of ToM capabilities in LLMs. We show that
FANToM is challenging for state-of-the-art LLMs, which perform significantly
worse than humans even with chain-of-thought reasoning or fine-tuning.

Computation and Language;Artificial Intelligence