title: Be like a Goldfish, Don't Memorize! Mitigating Memorization in
    Generative LLMs
summary: Large language models can memorize and repeat their training data, causing
  privacy and copyright risks. To mitigate memorization, we introduce a subtle
  modification to the next-token training objective that we call the goldfish
  loss. During training, a randomly sampled subset of tokens are excluded from
  the loss computation. These dropped tokens are not memorized by the model,
  which prevents verbatim reproduction of a complete chain of tokens from the
  training set. We run extensive experiments training billion-scale Llama-2
  models, both pre-trained and trained from scratch, and demonstrate significant
  reductions in extractable memorization with little to no impact on downstream
  benchmarks.
paper_authors._id: 6670da301b7f558a303a9061
paper_authors.name: Abhimanyu Hans
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:39:21.524Z
paper_authors._id: 6489c06c08b6a836a6dbd6aa
paper_authors.avatarUrl: /avatars/6ed1928992fc9d24bad0f12ebd2d84fe.svg
paper_authors.fullname: Abhimanyu Hans
paper_authors.type: user
paper_authors.user: ahans1
paper_authors._id: 6670da301b7f558a303a9062
paper_authors.name: Yuxin Wen
paper_authors._id: 6670da301b7f558a303a9063
paper_authors.name: Neel Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:18:05.872Z
paper_authors._id: 63e2b1ec282ee5f9624cfbcb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e2b1ec282ee5f9624cfbcb/4SVTp93cvRevacoJgiXzS.jpeg
paper_authors.fullname: Neel Jain
paper_authors.type: user
paper_authors.user: nsjain
paper_authors._id: 6670da301b7f558a303a9064
paper_authors.name: John Kirchenbauer
paper_authors._id: 6670da301b7f558a303a9065
paper_authors.name: Hamid Kazemi
paper_authors._id: 6670da301b7f558a303a9066
paper_authors.name: Prajwal Singhania
paper_authors._id: 6670da301b7f558a303a9067
paper_authors.name: Siddharth Singh
paper_authors._id: 6670da301b7f558a303a9068
paper_authors.name: Gowthami Somepalli
paper_authors._id: 6670da301b7f558a303a9069
paper_authors.name: Jonas Geiping
paper_authors._id: 6670da301b7f558a303a906a
paper_authors.name: Abhinav Bhatele
paper_authors._id: 6670da301b7f558a303a906b
paper_authors.name: Tom Goldstein
paper_title: Be like a Goldfish, Don't Memorize! Mitigating Memorization in
    Generative LLMs
paper_summary: Large language models can memorize and repeat their training data, causing
  privacy and copyright risks. To mitigate memorization, we introduce a subtle
  modification to the next-token training objective that we call the goldfish
  loss. During training, a randomly sampled subset of tokens are excluded from
  the loss computation. These dropped tokens are not memorized by the model,
  which prevents verbatim reproduction of a complete chain of tokens from the
  training set. We run extensive experiments training billion-scale Llama-2
  models, both pre-trained and trained from scratch, and demonstrate significant
  reductions in extractable memorization with little to no impact on downstream
  benchmarks.
paper_ai_keywords: goldfish loss next-token training objective training objective randomly sampled subset loss computation dropped tokens model memorization verbatim reproduction training set perplexity downstream benchmarks billion-scale Llama-2 models pre-trained models trained from scratch