LLMs achieve adult human performance on higher-order theory of mind
  tasks
This paper examines the extent to which large language models (LLMs) have
developed higher-order theory of mind (ToM); the human ability to reason about
multiple mental and emotional states in a recursive manner (e.g. I think that
you believe that she knows). This paper builds on prior work by introducing a
handwritten test suite -- Multi-Order Theory of Mind Q&A -- and using it to
compare the performance of five LLMs to a newly gathered adult human benchmark.
We find that GPT-4 and Flan-PaLM reach adult-level and near adult-level
performance on ToM tasks overall, and that GPT-4 exceeds adult performance on
6th order inferences. Our results suggest that there is an interplay between
model size and finetuning for the realisation of ToM abilities, and that the
best-performing LLMs have developed a generalised capacity for ToM. Given the
role that higher-order ToM plays in a wide range of cooperative and competitive
human behaviours, these findings have significant implications for user-facing
LLM applications.
665890797c3a646062755f13
Winnie Street
665890797c3a646062755f14
John Oliver Siy
665890797c3a646062755f15
Geoff Keeling
665890797c3a646062755f16
Adrien Baranes
665890797c3a646062755f17
Benjamin Barnett
665890797c3a646062755f18
Michael McKibben
665890797c3a646062755f19
Tatenda Kanyere
admin_assigned
2024-05-31T10:38:35.755Z
654a4a99f39b362cffb2e038
/avatars/507a5308de18f51e041e525d63899340.svg
Tatenda Kanyere
user
09tkanyerw
665890797c3a646062755f1a
Alison Lentz
665890797c3a646062755f1b
Blaise Aguera y Arcas
665890797c3a646062755f1c
Robin I. M. Dunbar
LLMs achieve adult human performance on higher-order theory of mind
  tasks
This paper examines the extent to which large language models (LLMs) have
developed higher-order theory of mind (ToM); the human ability to reason about
multiple mental and emotional states in a recursive manner (e.g. I think that
you believe that she knows). This paper builds on prior work by introducing a
handwritten test suite -- Multi-Order Theory of Mind Q&A -- and using it to
compare the performance of five LLMs to a newly gathered adult human benchmark.
We find that GPT-4 and Flan-PaLM reach adult-level and near adult-level
performance on ToM tasks overall, and that GPT-4 exceeds adult performance on
6th order inferences. Our results suggest that there is an interplay between
model size and finetuning for the realisation of ToM abilities, and that the
best-performing LLMs have developed a generalised capacity for ToM. Given the
role that higher-order ToM plays in a wide range of cooperative and competitive
human behaviours, these findings have significant implications for user-facing
LLM applications.
Large language models exhibit varying levels of higher-order theory of mind, with some models surpassing adult performance on recursive mental and emotional reasoning tasks.
large language models theory of mind higher-order theory of mind recursive reasoning Multi-Order Theory of Mind Q&A model size fine-tuning cooperative behaviors competitive behaviors