title: LLMs achieve adult human performance on higher-order theory of mind
    tasks
summary: This paper examines the extent to which large language models (LLMs) have
  developed higher-order theory of mind (ToM); the human ability to reason about
  multiple mental and emotional states in a recursive manner (e.g. I think that
  you believe that she knows). This paper builds on prior work by introducing a
  handwritten test suite -- Multi-Order Theory of Mind Q&A -- and using it to
  compare the performance of five LLMs to a newly gathered adult human benchmark.
  We find that GPT-4 and Flan-PaLM reach adult-level and near adult-level
  performance on ToM tasks overall, and that GPT-4 exceeds adult performance on
  6th order inferences. Our results suggest that there is an interplay between
  model size and finetuning for the realisation of ToM abilities, and that the
  best-performing LLMs have developed a generalised capacity for ToM. Given the
  role that higher-order ToM plays in a wide range of cooperative and competitive
  human behaviours, these findings have significant implications for user-facing
  LLM applications.
paper_authors._id: 665890797c3a646062755f13
paper_authors.name: Winnie Street
paper_authors._id: 665890797c3a646062755f14
paper_authors.name: John Oliver Siy
paper_authors._id: 665890797c3a646062755f15
paper_authors.name: Geoff Keeling
paper_authors._id: 665890797c3a646062755f16
paper_authors.name: Adrien Baranes
paper_authors._id: 665890797c3a646062755f17
paper_authors.name: Benjamin Barnett
paper_authors._id: 665890797c3a646062755f18
paper_authors.name: Michael McKibben
paper_authors._id: 665890797c3a646062755f19
paper_authors.name: Tatenda Kanyere
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:38:35.755Z
paper_authors._id: 654a4a99f39b362cffb2e038
paper_authors.avatarUrl: /avatars/507a5308de18f51e041e525d63899340.svg
paper_authors.fullname: Tatenda Kanyere
paper_authors.type: user
paper_authors.user: 09tkanyerw
paper_authors._id: 665890797c3a646062755f1a
paper_authors.name: Alison Lentz
paper_authors._id: 665890797c3a646062755f1b
paper_authors.name: Blaise Aguera y Arcas
paper_authors._id: 665890797c3a646062755f1c
paper_authors.name: Robin I. M. Dunbar
paper_title: LLMs achieve adult human performance on higher-order theory of mind
    tasks
paper_summary: This paper examines the extent to which large language models (LLMs) have
  developed higher-order theory of mind (ToM); the human ability to reason about
  multiple mental and emotional states in a recursive manner (e.g. I think that
  you believe that she knows). This paper builds on prior work by introducing a
  handwritten test suite -- Multi-Order Theory of Mind Q&A -- and using it to
  compare the performance of five LLMs to a newly gathered adult human benchmark.
  We find that GPT-4 and Flan-PaLM reach adult-level and near adult-level
  performance on ToM tasks overall, and that GPT-4 exceeds adult performance on
  6th order inferences. Our results suggest that there is an interplay between
  model size and finetuning for the realisation of ToM abilities, and that the
  best-performing LLMs have developed a generalised capacity for ToM. Given the
  role that higher-order ToM plays in a wide range of cooperative and competitive
  human behaviours, these findings have significant implications for user-facing
  LLM applications.
paper_ai_summary: Large language models exhibit varying levels of higher-order theory of mind, with some models surpassing adult performance on recursive mental and emotional reasoning tasks.
paper_ai_keywords: large language models theory of mind higher-order theory of mind recursive reasoning Multi-Order Theory of Mind Q&A model size fine-tuning cooperative behaviors competitive behaviors