Papers
arxiv:2406.16254

Confidence Regulation Neurons in Language Models

Published on Jun 24
· Submitted by gsarti on Jun 25
Authors:
,
,

Abstract

Despite their widespread use, the mechanisms by which large language models (LLMs) represent and regulate uncertainty in next-token predictions remain largely unexplored. This study investigates two critical components believed to influence this uncertainty: the recently discovered entropy neurons and a new set of components that we term token frequency neurons. Entropy neurons are characterized by an unusually high weight norm and influence the final layer normalization (LayerNorm) scale to effectively scale down the logits. Our work shows that entropy neurons operate by writing onto an unembedding null space, allowing them to impact the residual stream norm with minimal direct effect on the logits themselves. We observe the presence of entropy neurons across a range of models, up to 7 billion parameters. On the other hand, token frequency neurons, which we discover and describe here for the first time, boost or suppress each token's logit proportionally to its log frequency, thereby shifting the output distribution towards or away from the unigram distribution. Finally, we present a detailed case study where entropy neurons actively manage confidence in the setting of induction, i.e. detecting and continuing repeated subsequences.

Community

Paper submitter
edited Jun 25

New in the Daily Picks in Interpretability & Analysis of LMs (https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9)

This work focuses on neuron-level mechanisms for confidence calibration in LLMs, identifying entropy and token frequency neurons. Entropy neurons have high weight norms but minimal direct impact on logits. They write onto an unembedding null space and leverage layer normalization to scale down logits, effectively modulating output distribution entropy. Token frequency neurons boost or suppress token logits proportionally to their frequency, shifting the output distribution towards or away from the unigram distribution. Experiments demonstrate the interaction between entropy neurons and other mechanisms, such as induction heads, resulting in improved model performances thanks to the calibration of model outputs across the entire distribution.

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2406.16254 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2406.16254 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2406.16254 in a Space README.md to link it from this page.

Collections including this paper 5