Benchmarking Mental State Representations in Language Models
While numerous works have assessed the generative performance of language
models (LMs) on tasks requiring Theory of Mind reasoning, research into the
models' internal representation of mental states remains limited. Recent work
has used probing to demonstrate that LMs can represent beliefs of themselves
and others. However, these claims are accompanied by limited evaluation, making
it difficult to assess how mental state representations are affected by model
design and training choices. We report an extensive benchmark with various LM
types with different model sizes, fine-tuning approaches, and prompt designs to
study the robustness of mental state representations and memorisation issues
within the probes. Our results show that the quality of models' internal
representations of the beliefs of others increases with model size and, more
crucially, with fine-tuning. We are the first to study how prompt variations
impact probing performance on theory of mind tasks. We demonstrate that models'
representations are sensitive to prompt variations, even when such variations
should be beneficial. Finally, we complement previous activation editing
experiments on Theory of Mind tasks and show that it is possible to improve
models' reasoning performance by steering their activations without the need to
train any probe.
667e925e8a819eb0d305bce8
Matteo Bortoletto
claimed_verified
2024-06-28T12:11:38.262Z
66261ff68fee5ba32455ca80
https://cdn-avatars.huggingface.co/v1/production/uploads/66261ff68fee5ba32455ca80/4BWSZjF7nqsFi3i5l40cB.jpeg
Matteo Bortoletto
user
mb22222
667e925e8a819eb0d305bce9
Constantin Ruhdorfer
claimed_verified
2024-06-28T13:21:19.749Z
667eb8adf6fa6fb03b3daddd
/avatars/df105c2adf10620a19c3ff85d06aa64e.svg
Constantin Ruhdorfer
user
ConstantinRuhdorfer
667e925e8a819eb0d305bcea
Lei Shi
667e925e8a819eb0d305bceb
Andreas Bulling
Benchmarking Mental State Representations in Language Models
While numerous works have assessed the generative performance of language
models (LMs) on tasks requiring Theory of Mind reasoning, research into the
models' internal representation of mental states remains limited. Recent work
has used probing to demonstrate that LMs can represent beliefs of themselves
and others. However, these claims are accompanied by limited evaluation, making
it difficult to assess how mental state representations are affected by model
design and training choices. We report an extensive benchmark with various LM
types with different model sizes, fine-tuning approaches, and prompt designs to
study the robustness of mental state representations and memorisation issues
within the probes. Our results show that the quality of models' internal
representations of the beliefs of others increases with model size and, more
crucially, with fine-tuning. We are the first to study how prompt variations
impact probing performance on theory of mind tasks. We demonstrate that models'
representations are sensitive to prompt variations, even when such variations
should be beneficial. Finally, we complement previous activation editing
experiments on Theory of Mind tasks and show that it is possible to improve
models' reasoning performance by steering their activations without the need to
train any probe.
Research examines how language models represent mental states, showing that larger models and fine-tuning improve accuracy, and prompt variations affect probing performance. Activation editing can also enhance reasoning without additional training.
language models Theory of Mind probing mental state representations model size fine-tuning prompt designs memomization issues activation editing