title: Localizing Paragraph Memorization in Language Models
summary: Can we localize the weights and mechanisms used by a language model to
  memorize and recite entire paragraphs of its training data? In this paper, we
  show that while memorization is spread across multiple layers and model
  components, gradients of memorized paragraphs have a distinguishable spatial
  pattern, being larger in lower model layers than gradients of non-memorized
  examples. Moreover, the memorized examples can be unlearned by fine-tuning only
  the high-gradient weights. We localize a low-layer attention head that appears
  to be especially involved in paragraph memorization. This head is predominantly
  focusing its attention on distinctive, rare tokens that are least frequent in a
  corpus-level unigram distribution. Next, we study how localized memorization is
  across the tokens in the prefix by perturbing tokens and measuring the caused
  change in the decoding. A few distinctive tokens early in a prefix can often
  corrupt the entire continuation. Overall, memorized continuations are not only
  harder to unlearn, but also to corrupt than non-memorized ones.
paper_authors._id: 660a43fff4ab651901c081f3
paper_authors.name: Niklas Stoehr
paper_authors._id: 660a43fff4ab651901c081f4
paper_authors.name: Mitchell Gordon
paper_authors._id: 660a43fff4ab651901c081f5
paper_authors.name: Chiyuan Zhang
paper_authors._id: 660a43fff4ab651901c081f6
paper_authors.name: Owen Lewis
paper_title: Localizing Paragraph Memorization in Language Models
paper_summary: Can we localize the weights and mechanisms used by a language model to
  memorize and recite entire paragraphs of its training data? In this paper, we
  show that while memorization is spread across multiple layers and model
  components, gradients of memorized paragraphs have a distinguishable spatial
  pattern, being larger in lower model layers than gradients of non-memorized
  examples. Moreover, the memorized examples can be unlearned by fine-tuning only
  the high-gradient weights. We localize a low-layer attention head that appears
  to be especially involved in paragraph memorization. This head is predominantly
  focusing its attention on distinctive, rare tokens that are least frequent in a
  corpus-level unigram distribution. Next, we study how localized memorization is
  across the tokens in the prefix by perturbing tokens and measuring the caused
  change in the decoding. A few distinctive tokens early in a prefix can often
  corrupt the entire continuation. Overall, memorized continuations are not only
  harder to unlearn, but also to corrupt than non-memorized ones.
paper_ai_summary: Memorized paragraphs in language models exhibit distinguishable gradient patterns and can be localized to specific low-layer attention heads and distinctive tokens.
paper_ai_keywords: language model memorization gradients paragraph memorization attention head distinctive tokens unigram distribution decoding