title: Do I Know This Entity? Knowledge Awareness and Hallucinations in
    Language Models
summary: Hallucinations in large language models are a widespread problem, yet the
  mechanisms behind whether models will hallucinate are poorly understood,
  limiting our ability to solve this problem. Using sparse autoencoders as an
  interpretability tool, we discover that a key part of these mechanisms is
  entity recognition, where the model detects if an entity is one it can recall
  facts about. Sparse autoencoders uncover meaningful directions in the
  representation space, these detect whether the model recognizes an entity, e.g.
  detecting it doesn't know about an athlete or a movie. This suggests that
  models can have self-knowledge: internal representations about their own
  capabilities. These directions are causally relevant: capable of steering the
  model to refuse to answer questions about known entities, or to hallucinate
  attributes of unknown entities when it would otherwise refuse. We demonstrate
  that despite the sparse autoencoders being trained on the base model, these
  directions have a causal effect on the chat model's refusal behavior,
  suggesting that chat finetuning has repurposed this existing mechanism.
  Furthermore, we provide an initial exploration into the mechanistic role of
  these directions in the model, finding that they disrupt the attention of
  downstream heads that typically move entity attributes to the final token.
paper_authors._id: 67404b754feb9ca7e53aae25
paper_authors.name: Javier Ferrando
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:19:27.390Z
paper_authors._id: 61a5051ecd3eb3043f38e443
paper_authors.avatarUrl: /avatars/bcc94a31fab7486ca9d018245a289fb0.svg
paper_authors.fullname: Javier Ferrando
paper_authors.type: user
paper_authors.user: javifer
paper_authors._id: 67404b754feb9ca7e53aae26
paper_authors.name: Oscar Obeso
paper_authors._id: 67404b754feb9ca7e53aae27
paper_authors.name: Senthooran Rajamanoharan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:19:56.405Z
paper_authors._id: 6477122c99a5ce743ccf2f55
paper_authors.avatarUrl: /avatars/dee617374fc609e07eba3bcb2cd16810.svg
paper_authors.fullname: Senthooran Rajamanoharan
paper_authors.type: user
paper_authors.user: srdm
paper_authors._id: 67404b754feb9ca7e53aae28
paper_authors.name: Neel Nanda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:20:12.295Z
paper_authors._id: 62669380c8bc5cf80ca97350
paper_authors.avatarUrl: /avatars/6d5cd2261163308b82341c1ce28984d1.svg
paper_authors.fullname: Neel Nanda
paper_authors.type: user
paper_authors.user: NeelNanda
paper_title: Do I Know This Entity? Knowledge Awareness and Hallucinations in
    Language Models
paper_summary: Hallucinations in large language models are a widespread problem, yet the
  mechanisms behind whether models will hallucinate are poorly understood,
  limiting our ability to solve this problem. Using sparse autoencoders as an
  interpretability tool, we discover that a key part of these mechanisms is
  entity recognition, where the model detects if an entity is one it can recall
  facts about. Sparse autoencoders uncover meaningful directions in the
  representation space, these detect whether the model recognizes an entity, e.g.
  detecting it doesn't know about an athlete or a movie. This suggests that
  models can have self-knowledge: internal representations about their own
  capabilities. These directions are causally relevant: capable of steering the
  model to refuse to answer questions about known entities, or to hallucinate
  attributes of unknown entities when it would otherwise refuse. We demonstrate
  that despite the sparse autoencoders being trained on the base model, these
  directions have a causal effect on the chat model's refusal behavior,
  suggesting that chat finetuning has repurposed this existing mechanism.
  Furthermore, we provide an initial exploration into the mechanistic role of
  these directions in the model, finding that they disrupt the attention of
  downstream heads that typically move entity attributes to the final token.
paper_ai_summary: Sparse autoencoders reveal that large language models can recognize known entities and have internal self-knowledge about their capabilities, influencing their tendency to hallucinate.
paper_ai_keywords: sparse autoencoders entity recognition representation space self-knowledge causal effect chat finetuning attention entity attributes