title: Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs
summary: LLMs process text as sequences of tokens that roughly correspond to words,
  where less common words are represented by multiple tokens. However, individual
  tokens are often semantically unrelated to the meanings of the words/concepts
  they comprise. For example, Llama-2-7b's tokenizer splits the word
  "northeastern" into the tokens ['_n', 'ort', 'he', 'astern'], none of which
  correspond to semantically meaningful units like "north" or "east." Similarly,
  the overall meanings of named entities like "Neil Young" and multi-word
  expressions like "break a leg" cannot be directly inferred from their
  constituent tokens. Mechanistically, how do LLMs convert such arbitrary groups
  of tokens into useful higher-level representations? In this work, we find that
  last token representations of named entities and multi-token words exhibit a
  pronounced "erasure" effect, where information about previous and current
  tokens is rapidly forgotten in early layers. Using this observation, we propose
  a method to "read out" the implicit vocabulary of an autoregressive LLM by
  examining differences in token representations across layers, and present
  results of this method for Llama-2-7b and Llama-3-8B. To our knowledge, this is
  the first attempt to probe the implicit vocabulary of an LLM.
paper_authors._id: 668420ae3b6f1b196d0074ef
paper_authors.name: Sheridan Feucht
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:06:30.903Z
paper_authors._id: 653bf2002c81c4adc8f99d70
paper_authors.avatarUrl: /avatars/0bd74739cef2df5d4889f108de17a120.svg
paper_authors.fullname: Sheridan Feucht
paper_authors.type: user
paper_authors.user: sfeucht
paper_authors._id: 668420ae3b6f1b196d0074f0
paper_authors.name: David Atkinson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:06:36.877Z
paper_authors._id: 64769e3c3d8d1cb791909f09
paper_authors.avatarUrl: /avatars/c09f6f1df52a710db6a65e37a4545458.svg
paper_authors.fullname: David Atkinson
paper_authors.type: user
paper_authors.user: datkinson
paper_authors._id: 668420ae3b6f1b196d0074f1
paper_authors.name: Byron Wallace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:06:43.376Z
paper_authors._id: 6474aa4433192631bace7ff0
paper_authors.avatarUrl: /avatars/06b586fc65350800a9aeffacd7bbdf50.svg
paper_authors.fullname: Byron Wallace
paper_authors.type: user
paper_authors.user: byroncwallace
paper_authors._id: 668420ae3b6f1b196d0074f2
paper_authors.name: David Bau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:06:51.724Z
paper_authors._id: 6214d6c01e35c843d42d1f77
paper_authors.avatarUrl: /avatars/ac208cd180b4f3ed1ec367e581facfcf.svg
paper_authors.fullname: David Bau
paper_authors.type: user
paper_authors.user: davidbau
paper_title: Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs
paper_summary: LLMs process text as sequences of tokens that roughly correspond to words,
  where less common words are represented by multiple tokens. However, individual
  tokens are often semantically unrelated to the meanings of the words/concepts
  they comprise. For example, Llama-2-7b's tokenizer splits the word
  "northeastern" into the tokens ['_n', 'ort', 'he', 'astern'], none of which
  correspond to semantically meaningful units like "north" or "east." Similarly,
  the overall meanings of named entities like "Neil Young" and multi-word
  expressions like "break a leg" cannot be directly inferred from their
  constituent tokens. Mechanistically, how do LLMs convert such arbitrary groups
  of tokens into useful higher-level representations? In this work, we find that
  last token representations of named entities and multi-token words exhibit a
  pronounced "erasure" effect, where information about previous and current
  tokens is rapidly forgotten in early layers. Using this observation, we propose
  a method to "read out" the implicit vocabulary of an autoregressive LLM by
  examining differences in token representations across layers, and present
  results of this method for Llama-2-7b and Llama-3-8B. To our knowledge, this is
  the first attempt to probe the implicit vocabulary of an LLM.
paper_ai_summary: LLMs convert token sequences into higher-level representations by exhibiting an "erasure" effect in early layers, where information is rapidly forgotten; a method is proposed to probe the implicit vocabulary of LLMs by analyzing token representations across layers.
paper_ai_keywords: LLMs autoregressive LLMs token representations erasure effect implicit vocabulary