title: Memorization-Compression Cycles Improve Generalization
summary: We prove theoretically that generalization improves not only through data
  scaling but also by compressing internal representations. To operationalize
  this insight, we introduce the Information Bottleneck Language Modeling (IBLM)
  objective, which reframes language modeling as a constrained optimization
  problem: minimizing representation entropy subject to optimal prediction
  performance. Empirically, we observe an emergent memorization-compression cycle
  during LLM pretraining, evidenced by oscillation positive/negative gradient
  alignment between cross-entropy and Matrix-Based Entropy (MBE), a measure of
  representation entropy. This pattern closely mirrors the predictive-compressive
  trade-off prescribed by IBLM and also parallels the biological alternation
  between awake learning and sleep consolidation. Motivated by this observation,
  we propose Gated Phase Transition (GAPT), a training algorithm that adaptively
  switches between memorization and compression phases. When applied to GPT-2
  pretraining on FineWeb dataset, GAPT reduces MBE by 50% and improves
  cross-entropy by 4.8%. GAPT improves OOD generalizatino by 35% in a pretraining
  task on arithmetic multiplication. In a setting designed to simulate
  catastrophic forgetting, GAPT reduces interference by compressing and
  separating representations, achieving a 97% improvement in separation -
  paralleling the functional role of sleep consolidation.
paper_authors._id: 6824392a5d39467d53a7ca9d
paper_authors.name: Fangyuan Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:33:49.661Z
paper_authors._id: 64d98ef7a4839890b25eb78b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d98ef7a4839890b25eb78b/215-CSVLl81z6CAq0ECWU.jpeg
paper_authors.fullname: Fangyuan Yu
paper_authors.type: user
paper_authors.user: Ksgk-fy
paper_title: Memorization-Compression Cycles Improve Generalization
paper_summary: We prove theoretically that generalization improves not only through data
  scaling but also by compressing internal representations. To operationalize
  this insight, we introduce the Information Bottleneck Language Modeling (IBLM)
  objective, which reframes language modeling as a constrained optimization
  problem: minimizing representation entropy subject to optimal prediction
  performance. Empirically, we observe an emergent memorization-compression cycle
  during LLM pretraining, evidenced by oscillation positive/negative gradient
  alignment between cross-entropy and Matrix-Based Entropy (MBE), a measure of
  representation entropy. This pattern closely mirrors the predictive-compressive
  trade-off prescribed by IBLM and also parallels the biological alternation
  between awake learning and sleep consolidation. Motivated by this observation,
  we propose Gated Phase Transition (GAPT), a training algorithm that adaptively
  switches between memorization and compression phases. When applied to GPT-2
  pretraining on FineWeb dataset, GAPT reduces MBE by 50% and improves
  cross-entropy by 4.8%. GAPT improves OOD generalizatino by 35% in a pretraining
  task on arithmetic multiplication. In a setting designed to simulate
  catastrophic forgetting, GAPT reduces interference by compressing and
  separating representations, achieving a 97% improvement in separation -
  paralleling the functional role of sleep consolidation.
paper_ai_summary: Information Bottleneck Language Modeling introduces a reversible memorization-compression cycle in language model training to enhance generalization and reduce overfitting.
paper_ai_keywords: Information Bottleneck Language Modeling IBLM constrained optimization representation entropy Matrix-Based Entropy MBE Gated Phase Transition GAPT FineWeb dataset OOD generalization catastrophic forgetting sleep consolidation