title: How new data permeates LLM knowledge and how to dilute it
summary: Large language models learn and continually learn through the accumulation of
  gradient-based updates, but how individual pieces of new information affect
  existing knowledge, leading to both beneficial generalization and problematic
  hallucination, remains poorly understood. We demonstrate that when learning new
  information, LLMs exhibit a "priming" effect: learning a new fact can cause the
  model to inappropriately apply that knowledge in unrelated contexts. To
  systematically study this phenomenon, we introduce "Outlandish," a carefully
  curated dataset of 1320 diverse text samples designed to probe how new
  knowledge permeates through an LLM's existing knowledge base. Using this
  dataset, we show that the degree of priming after learning new information can
  be predicted by measuring the token probability of key words before learning.
  This relationship holds robustly across different model architectures (PALM-2,
  Gemma, Llama), sizes, and training stages. Finally, we develop two novel
  techniques to modulate how new knowledge affects existing model behavior: (1) a
  ``stepping-stone'' text augmentation strategy and (2) an ``ignore-k'' update
  pruning method. These approaches reduce undesirable priming effects by 50-95\%
  while preserving the model's ability to learn new information. Our findings
  provide both empirical insights into how LLMs learn and practical tools for
  improving the specificity of knowledge insertion in language models. Further
  materials: https://sunchipsster1.github.io/projects/outlandish/
paper_authors._id: 67fe132f1d1bc292f7cdbc0f
paper_authors.name: Chen Sun
paper_authors._id: 67fe132f1d1bc292f7cdbc10
paper_authors.name: Renat Aksitov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:36:40.229Z
paper_authors._id: 6003c87f532970af8c4d3a4a
paper_authors.avatarUrl: /avatars/a4c5fbe427791d02e3cee208f22f18c4.svg
paper_authors.fullname: Renat Aksitov
paper_authors.type: user
paper_authors.user: mendor
paper_authors._id: 67fe132f1d1bc292f7cdbc11
paper_authors.name: Andrey Zhmoginov
paper_authors._id: 67fe132f1d1bc292f7cdbc12
paper_authors.name: Nolan Andrew Miller
paper_authors._id: 67fe132f1d1bc292f7cdbc13
paper_authors.name: Max Vladymyrov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:37:00.587Z
paper_authors._id: 6502fe0bb1792803da806d42
paper_authors.avatarUrl: /avatars/60280ce59f1f0d67e4210b0453039282.svg
paper_authors.fullname: Max Vladymyrov
paper_authors.type: user
paper_authors.user: gozzo87
paper_authors._id: 67fe132f1d1bc292f7cdbc14
paper_authors.name: Ulrich Rueckert
paper_authors._id: 67fe132f1d1bc292f7cdbc15
paper_authors.name: Been Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T09:23:17.092Z
paper_authors._id: 65dec807f4cf523519261714
paper_authors.avatarUrl: /avatars/c6acb4f0c7ad715ff2c821c85be88d73.svg
paper_authors.fullname: been kim
paper_authors.type: user
paper_authors.user: Beenkim
paper_authors._id: 67fe132f1d1bc292f7cdbc16
paper_authors.name: Mark Sandler
paper_title: How new data permeates LLM knowledge and how to dilute it
paper_summary: Large language models learn and continually learn through the accumulation of
  gradient-based updates, but how individual pieces of new information affect
  existing knowledge, leading to both beneficial generalization and problematic
  hallucination, remains poorly understood. We demonstrate that when learning new
  information, LLMs exhibit a "priming" effect: learning a new fact can cause the
  model to inappropriately apply that knowledge in unrelated contexts. To
  systematically study this phenomenon, we introduce "Outlandish," a carefully
  curated dataset of 1320 diverse text samples designed to probe how new
  knowledge permeates through an LLM's existing knowledge base. Using this
  dataset, we show that the degree of priming after learning new information can
  be predicted by measuring the token probability of key words before learning.
  This relationship holds robustly across different model architectures (PALM-2,
  Gemma, Llama), sizes, and training stages. Finally, we develop two novel
  techniques to modulate how new knowledge affects existing model behavior: (1) a
  ``stepping-stone'' text augmentation strategy and (2) an ``ignore-k'' update
  pruning method. These approaches reduce undesirable priming effects by 50-95\%
  while preserving the model's ability to learn new information. Our findings
  provide both empirical insights into how LLMs learn and practical tools for
  improving the specificity of knowledge insertion in language models. Further
  materials: https://sunchipsster1.github.io/projects/outlandish/
paper_ai_summary: LLMs exhibit priming effects when learning new information, which can lead to inappropriate knowledge application; the degree of priming is predictable and modifiable through specific techniques.
paper_ai_keywords: gradient-based updates large language models priming effect Outlandish dataset token probability PALM-2 Gemma Llama stepping-stone text augmentation ignore-k update pruning