title: Benchmarking Mental State Representations in Language Models
summary: While numerous works have assessed the generative performance of language
  models (LMs) on tasks requiring Theory of Mind reasoning, research into the
  models' internal representation of mental states remains limited. Recent work
  has used probing to demonstrate that LMs can represent beliefs of themselves
  and others. However, these claims are accompanied by limited evaluation, making
  it difficult to assess how mental state representations are affected by model
  design and training choices. We report an extensive benchmark with various LM
  types with different model sizes, fine-tuning approaches, and prompt designs to
  study the robustness of mental state representations and memorisation issues
  within the probes. Our results show that the quality of models' internal
  representations of the beliefs of others increases with model size and, more
  crucially, with fine-tuning. We are the first to study how prompt variations
  impact probing performance on theory of mind tasks. We demonstrate that models'
  representations are sensitive to prompt variations, even when such variations
  should be beneficial. Finally, we complement previous activation editing
  experiments on Theory of Mind tasks and show that it is possible to improve
  models' reasoning performance by steering their activations without the need to
  train any probe.
paper_authors._id: 667e925e8a819eb0d305bce8
paper_authors.name: Matteo Bortoletto
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T12:11:38.262Z
paper_authors._id: 66261ff68fee5ba32455ca80
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66261ff68fee5ba32455ca80/4BWSZjF7nqsFi3i5l40cB.jpeg
paper_authors.fullname: Matteo Bortoletto
paper_authors.type: user
paper_authors.user: mb22222
paper_authors._id: 667e925e8a819eb0d305bce9
paper_authors.name: Constantin Ruhdorfer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T13:21:19.749Z
paper_authors._id: 667eb8adf6fa6fb03b3daddd
paper_authors.avatarUrl: /avatars/df105c2adf10620a19c3ff85d06aa64e.svg
paper_authors.fullname: Constantin Ruhdorfer
paper_authors.type: user
paper_authors.user: ConstantinRuhdorfer
paper_authors._id: 667e925e8a819eb0d305bcea
paper_authors.name: Lei Shi
paper_authors._id: 667e925e8a819eb0d305bceb
paper_authors.name: Andreas Bulling
paper_title: Benchmarking Mental State Representations in Language Models
paper_summary: While numerous works have assessed the generative performance of language
  models (LMs) on tasks requiring Theory of Mind reasoning, research into the
  models' internal representation of mental states remains limited. Recent work
  has used probing to demonstrate that LMs can represent beliefs of themselves
  and others. However, these claims are accompanied by limited evaluation, making
  it difficult to assess how mental state representations are affected by model
  design and training choices. We report an extensive benchmark with various LM
  types with different model sizes, fine-tuning approaches, and prompt designs to
  study the robustness of mental state representations and memorisation issues
  within the probes. Our results show that the quality of models' internal
  representations of the beliefs of others increases with model size and, more
  crucially, with fine-tuning. We are the first to study how prompt variations
  impact probing performance on theory of mind tasks. We demonstrate that models'
  representations are sensitive to prompt variations, even when such variations
  should be beneficial. Finally, we complement previous activation editing
  experiments on Theory of Mind tasks and show that it is possible to improve
  models' reasoning performance by steering their activations without the need to
  train any probe.
paper_ai_keywords: language models (LMs) Theory of Mind reasoning probing mental states representations fine-tuning prompt designs memorisation issues robustness model size activation editing reasoning performance steering activations