title: Neurons in Large Language Models: Dead, N-gram, Positional
summary: We analyze a family of large language models in such a lightweight manner
  that can be done on a single GPU. Specifically, we focus on the OPT family of
  models ranging from 125m to 66b parameters and rely only on whether an FFN
  neuron is activated or not. First, we find that the early part of the network
  is sparse and represents many discrete features. Here, many neurons (more than
  70% in some layers of the 66b model) are "dead", i.e. they never activate on a
  large collection of diverse data. At the same time, many of the alive neurons
  are reserved for discrete features and act as token and n-gram detectors.
  Interestingly, their corresponding FFN updates not only promote next token
  candidates as could be expected, but also explicitly focus on removing the
  information about triggering them tokens, i.e., current input. To the best of
  our knowledge, this is the first example of mechanisms specialized at removing
  (rather than adding) information from the residual stream. With scale, models
  become more sparse in a sense that they have more dead neurons and token
  detectors. Finally, some neurons are positional: them being activated or not
  depends largely (or solely) on position and less so (or not at all) on textual
  data. We find that smaller models have sets of neurons acting as position range
  indicators while larger models operate in a less explicit manner.
paper_authors._id: 64ffcd416a230e55a6537377
paper_authors.name: Elena Voita
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-13T13:19:21.521Z
paper_authors._id: 65018065c56ace4fdc0014b4
paper_authors.avatarUrl: /avatars/d5e02ca2e88bb9b17f92703927a34df4.svg
paper_authors.fullname: Elena Voita
paper_authors.type: user
paper_authors.user: lena-voita
paper_authors._id: 64ffcd416a230e55a6537378
paper_authors.name: Javier Ferrando
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T11:07:34.873Z
paper_authors._id: 61a5051ecd3eb3043f38e443
paper_authors.avatarUrl: /avatars/bcc94a31fab7486ca9d018245a289fb0.svg
paper_authors.fullname: Javier Ferrando
paper_authors.type: user
paper_authors.user: javifer
paper_authors._id: 64ffcd416a230e55a6537379
paper_authors.name: Christoforos Nalmpantis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T11:07:42.253Z
paper_authors._id: 627e1e4f9632ca4815f0a729
paper_authors.avatarUrl: /avatars/2ba615cdc1d9e3eb0397366cff7a7c8b.svg
paper_authors.fullname: Christoforos Nalmpantis
paper_authors.type: user
paper_authors.user: christofernal
paper_title: Neurons in Large Language Models: Dead, N-gram, Positional
paper_summary: We analyze a family of large language models in such a lightweight manner
  that can be done on a single GPU. Specifically, we focus on the OPT family of
  models ranging from 125m to 66b parameters and rely only on whether an FFN
  neuron is activated or not. First, we find that the early part of the network
  is sparse and represents many discrete features. Here, many neurons (more than
  70% in some layers of the 66b model) are "dead", i.e. they never activate on a
  large collection of diverse data. At the same time, many of the alive neurons
  are reserved for discrete features and act as token and n-gram detectors.
  Interestingly, their corresponding FFN updates not only promote next token
  candidates as could be expected, but also explicitly focus on removing the
  information about triggering them tokens, i.e., current input. To the best of
  our knowledge, this is the first example of mechanisms specialized at removing
  (rather than adding) information from the residual stream. With scale, models
  become more sparse in a sense that they have more dead neurons and token
  detectors. Finally, some neurons are positional: them being activated or not
  depends largely (or solely) on position and less so (or not at all) on textual
  data. We find that smaller models have sets of neurons acting as position range
  indicators while larger models operate in a less explicit manner.
paper_ai_keywords: neural fields OPT family FFN (Feedforward Neural Network) dead neurons discrete features token detectors n-gram detectors residual stream position range indicators