title: Confidence Regulation Neurons in Language Models
summary: Despite their widespread use, the mechanisms by which large language models
  (LLMs) represent and regulate uncertainty in next-token predictions remain
  largely unexplored. This study investigates two critical components believed to
  influence this uncertainty: the recently discovered entropy neurons and a new
  set of components that we term token frequency neurons. Entropy neurons are
  characterized by an unusually high weight norm and influence the final layer
  normalization (LayerNorm) scale to effectively scale down the logits. Our work
  shows that entropy neurons operate by writing onto an unembedding null space,
  allowing them to impact the residual stream norm with minimal direct effect on
  the logits themselves. We observe the presence of entropy neurons across a
  range of models, up to 7 billion parameters. On the other hand, token frequency
  neurons, which we discover and describe here for the first time, boost or
  suppress each token's logit proportionally to its log frequency, thereby
  shifting the output distribution towards or away from the unigram distribution.
  Finally, we present a detailed case study where entropy neurons actively manage
  confidence in the setting of induction, i.e. detecting and continuing repeated
  subsequences.
paper_authors._id: 667a74f59f501609d28e34b9
paper_authors.name: Alessandro Stolfo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:38:41.156Z
paper_authors._id: 64c7dda1d418013c77d3acec
paper_authors.avatarUrl: /avatars/10bc3e8754e1a9815da57d3aec2f0bb3.svg
paper_authors.fullname: Alessandro Stolfo
paper_authors.type: user
paper_authors.user: alestolfo
paper_authors._id: 667a74f59f501609d28e34ba
paper_authors.name: Ben Wu
paper_authors._id: 667a74f59f501609d28e34bb
paper_authors.name: Wes Gurnee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:39:02.040Z
paper_authors._id: 631f446fdaa9591e52319035
paper_authors.avatarUrl: /avatars/34d7ff8bc59aac0ed77a961596c4e4b1.svg
paper_authors.fullname: Wes Gurnee
paper_authors.type: user
paper_authors.user: wesg
paper_authors._id: 667a74f59f501609d28e34bc
paper_authors.name: Yonatan Belinkov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:39:07.898Z
paper_authors._id: 614c57f1ee44bcfe57b366d6
paper_authors.avatarUrl: /avatars/186a9aed84681246f48ed2a012c50def.svg
paper_authors.fullname: Yonatan Belinkov
paper_authors.type: user
paper_authors.user: belinkov
paper_authors._id: 667a74f59f501609d28e34bd
paper_authors.name: Xingyi Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:39:17.590Z
paper_authors._id: 62e795ac7c106973ac6515eb
paper_authors.avatarUrl: /avatars/2eaa80217aae2a8dbd6cc350b7f35311.svg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: Xingyi
paper_authors._id: 667a74f59f501609d28e34be
paper_authors.name: Mrinmaya Sachan
paper_authors._id: 667a74f59f501609d28e34bf
paper_authors.name: Neel Nanda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:39:26.400Z
paper_authors._id: 62669380c8bc5cf80ca97350
paper_authors.avatarUrl: /avatars/6d5cd2261163308b82341c1ce28984d1.svg
paper_authors.fullname: Neel Nanda
paper_authors.type: user
paper_authors.user: NeelNanda
paper_title: Confidence Regulation Neurons in Language Models
paper_summary: Despite their widespread use, the mechanisms by which large language models
  (LLMs) represent and regulate uncertainty in next-token predictions remain
  largely unexplored. This study investigates two critical components believed to
  influence this uncertainty: the recently discovered entropy neurons and a new
  set of components that we term token frequency neurons. Entropy neurons are
  characterized by an unusually high weight norm and influence the final layer
  normalization (LayerNorm) scale to effectively scale down the logits. Our work
  shows that entropy neurons operate by writing onto an unembedding null space,
  allowing them to impact the residual stream norm with minimal direct effect on
  the logits themselves. We observe the presence of entropy neurons across a
  range of models, up to 7 billion parameters. On the other hand, token frequency
  neurons, which we discover and describe here for the first time, boost or
  suppress each token's logit proportionally to its log frequency, thereby
  shifting the output distribution towards or away from the unigram distribution.
  Finally, we present a detailed case study where entropy neurons actively manage
  confidence in the setting of induction, i.e. detecting and continuing repeated
  subsequences.
paper_ai_summary: Entropy neurons and token frequency neurons in large language models influence uncertainty by scaling down logits and adjusting output distributions based on token frequency, respectively.
paper_ai_keywords: large language models entropy neurons token frequency neurons normalization logits unembedding null space residual stream norm unigram distribution induction