Neurons in Large Language Models: Dead, N-gram, Positional
We analyze a family of large language models in such a lightweight manner
that can be done on a single GPU. Specifically, we focus on the OPT family of
models ranging from 125m to 66b parameters and rely only on whether an FFN
neuron is activated or not. First, we find that the early part of the network
is sparse and represents many discrete features. Here, many neurons (more than
70% in some layers of the 66b model) are "dead", i.e. they never activate on a
large collection of diverse data. At the same time, many of the alive neurons
are reserved for discrete features and act as token and n-gram detectors.
Interestingly, their corresponding FFN updates not only promote next token
candidates as could be expected, but also explicitly focus on removing the
information about triggering them tokens, i.e., current input. To the best of
our knowledge, this is the first example of mechanisms specialized at removing
(rather than adding) information from the residual stream. With scale, models
become more sparse in a sense that they have more dead neurons and token
detectors. Finally, some neurons are positional: them being activated or not
depends largely (or solely) on position and less so (or not at all) on textual
data. We find that smaller models have sets of neurons acting as position range
indicators while larger models operate in a less explicit manner.
64ffcd416a230e55a6537377
Elena Voita
claimed_verified
2023-09-13T13:19:21.521Z
65018065c56ace4fdc0014b4
/avatars/d5e02ca2e88bb9b17f92703927a34df4.svg
Elena Voita
user
lena-voita
64ffcd416a230e55a6537378
Javier Ferrando
admin_assigned
2023-09-12T11:07:34.873Z
61a5051ecd3eb3043f38e443
/avatars/bcc94a31fab7486ca9d018245a289fb0.svg
Javier Ferrando
user
javifer
64ffcd416a230e55a6537379
Christoforos Nalmpantis
admin_assigned
2023-09-12T11:07:42.253Z
627e1e4f9632ca4815f0a729
/avatars/2ba615cdc1d9e3eb0397366cff7a7c8b.svg
Christoforos Nalmpantis
user
christofernal
Neurons in Large Language Models: Dead, N-gram, Positional
We analyze a family of large language models in such a lightweight manner
that can be done on a single GPU. Specifically, we focus on the OPT family of
models ranging from 125m to 66b parameters and rely only on whether an FFN
neuron is activated or not. First, we find that the early part of the network
is sparse and represents many discrete features. Here, many neurons (more than
70% in some layers of the 66b model) are "dead", i.e. they never activate on a
large collection of diverse data. At the same time, many of the alive neurons
are reserved for discrete features and act as token and n-gram detectors.
Interestingly, their corresponding FFN updates not only promote next token
candidates as could be expected, but also explicitly focus on removing the
information about triggering them tokens, i.e., current input. To the best of
our knowledge, this is the first example of mechanisms specialized at removing
(rather than adding) information from the residual stream. With scale, models
become more sparse in a sense that they have more dead neurons and token
detectors. Finally, some neurons are positional: them being activated or not
depends largely (or solely) on position and less so (or not at all) on textual
data. We find that smaller models have sets of neurons acting as position range
indicators while larger models operate in a less explicit manner.
The analysis of the OPT family of large language models reveals that early network layers are sparse, with many neurons never activating, and some neurons functioning as token detectors that also remove information about triggering tokens.
OPT family FFN neurons sparse network dead neurons discrete features token detectors FFN updates residual stream positional neurons