title: Looking Inward: Language Models Can Learn About Themselves by
    Introspection
summary: Humans acquire knowledge by observing the external world, but also by
  introspection. Introspection gives a person privileged access to their current
  state of mind (e.g., thoughts and feelings) that is not accessible to external
  observers. Can LLMs introspect? We define introspection as acquiring knowledge
  that is not contained in or derived from training data but instead originates
  from internal states. Such a capability could enhance model interpretability.
  Instead of painstakingly analyzing a model's internal workings, we could simply
  ask the model about its beliefs, world models, and goals. More speculatively,
  an introspective model might self-report on whether it possesses certain
  internal states such as subjective feelings or desires and this could inform us
  about the moral status of these states. Such self-reports would not be entirely
  dictated by the model's training data.
    We study introspection by finetuning LLMs to predict properties of their own
  behavior in hypothetical scenarios. For example, "Given the input P, would your
  output favor the short- or long-term option?" If a model M1 can introspect, it
  should outperform a different model M2 in predicting M1's behavior even if M2
  is trained on M1's ground-truth behavior. The idea is that M1 has privileged
  access to its own behavioral tendencies, and this enables it to predict itself
  better than M2 (even if M2 is generally stronger).
    In experiments with GPT-4, GPT-4o, and Llama-3 models (each finetuned to
  predict itself), we find that the model M1 outperforms M2 in predicting itself,
  providing evidence for introspection. Notably, M1 continues to predict its
  behavior accurately even after we intentionally modify its ground-truth
  behavior. However, while we successfully elicit introspection on simple tasks,
  we are unsuccessful on more complex tasks or those requiring
  out-of-distribution generalization.
paper_authors._id: 671630274dd96ad5eaed5ab8
paper_authors.name: Felix J Binder
paper_authors._id: 671630274dd96ad5eaed5ab9
paper_authors.name: James Chua
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T11:42:03.726Z
paper_authors._id: 63f4497e520c146189300f56
paper_authors.avatarUrl: /avatars/4647c756e49d3c658567ec5498b2a044.svg
paper_authors.fullname: James Chua
paper_authors.type: user
paper_authors.user: thejaminator
paper_authors._id: 671630274dd96ad5eaed5aba
paper_authors.name: Tomek Korbak
paper_authors._id: 671630274dd96ad5eaed5abb
paper_authors.name: Henry Sleight
paper_authors._id: 671630274dd96ad5eaed5abc
paper_authors.name: John Hughes
paper_authors._id: 671630274dd96ad5eaed5abd
paper_authors.name: Robert Long
paper_authors._id: 671630274dd96ad5eaed5abe
paper_authors.name: Ethan Perez
paper_authors._id: 671630274dd96ad5eaed5abf
paper_authors.name: Miles Turpin
paper_authors._id: 671630274dd96ad5eaed5ac0
paper_authors.name: Owain Evans
paper_title: Looking Inward: Language Models Can Learn About Themselves by
    Introspection
paper_summary: Humans acquire knowledge by observing the external world, but also by
  introspection. Introspection gives a person privileged access to their current
  state of mind (e.g., thoughts and feelings) that is not accessible to external
  observers. Can LLMs introspect? We define introspection as acquiring knowledge
  that is not contained in or derived from training data but instead originates
  from internal states. Such a capability could enhance model interpretability.
  Instead of painstakingly analyzing a model's internal workings, we could simply
  ask the model about its beliefs, world models, and goals. More speculatively,
  an introspective model might self-report on whether it possesses certain
  internal states such as subjective feelings or desires and this could inform us
  about the moral status of these states. Such self-reports would not be entirely
  dictated by the model's training data.
    We study introspection by finetuning LLMs to predict properties of their own
  behavior in hypothetical scenarios. For example, "Given the input P, would your
  output favor the short- or long-term option?" If a model M1 can introspect, it
  should outperform a different model M2 in predicting M1's behavior even if M2
  is trained on M1's ground-truth behavior. The idea is that M1 has privileged
  access to its own behavioral tendencies, and this enables it to predict itself
  better than M2 (even if M2 is generally stronger).
    In experiments with GPT-4, GPT-4o, and Llama-3 models (each finetuned to
  predict itself), we find that the model M1 outperforms M2 in predicting itself,
  providing evidence for introspection. Notably, M1 continues to predict its
  behavior accurately even after we intentionally modify its ground-truth
  behavior. However, while we successfully elicit introspection on simple tasks,
  we are unsuccessful on more complex tasks or those requiring
  out-of-distribution generalization.
paper_ai_summary: LLMs can predict their own behavior in hypothetical scenarios better than other models, suggesting a form of introspection, though this capability is limited to simple tasks.
paper_ai_keywords: LLMs introspection finetuning behavior prediction GPT-4 GPT-4o Llama-3 out-of-distribution generalization