Memorization-Compression Cycles Improve Generalization
We prove theoretically that generalization improves not only through data
scaling but also by compressing internal representations. To operationalize
this insight, we introduce the Information Bottleneck Language Modeling (IBLM)
objective, which reframes language modeling as a constrained optimization
problem: minimizing representation entropy subject to optimal prediction
performance. Empirically, we observe an emergent memorization-compression cycle
during LLM pretraining, evidenced by oscillation positive/negative gradient
alignment between cross-entropy and Matrix-Based Entropy (MBE), a measure of
representation entropy. This pattern closely mirrors the predictive-compressive
trade-off prescribed by IBLM and also parallels the biological alternation
between awake learning and sleep consolidation. Motivated by this observation,
we propose Gated Phase Transition (GAPT), a training algorithm that adaptively
switches between memorization and compression phases. When applied to GPT-2
pretraining on FineWeb dataset, GAPT reduces MBE by 50% and improves
cross-entropy by 4.8%. GAPT improves OOD generalizatino by 35% in a pretraining
task on arithmetic multiplication. In a setting designed to simulate
catastrophic forgetting, GAPT reduces interference by compressing and
separating representations, achieving a 97% improvement in separation -
paralleling the functional role of sleep consolidation.
6824392a5d39467d53a7ca9d
Fangyuan Yu
claimed_verified
2025-05-14T07:33:49.661Z
64d98ef7a4839890b25eb78b
https://cdn-avatars.huggingface.co/v1/production/uploads/64d98ef7a4839890b25eb78b/215-CSVLl81z6CAq0ECWU.jpeg
Fangyuan Yu
user
Ksgk-fy
Memorization-Compression Cycles Improve Generalization
We prove theoretically that generalization improves not only through data
scaling but also by compressing internal representations. To operationalize
this insight, we introduce the Information Bottleneck Language Modeling (IBLM)
objective, which reframes language modeling as a constrained optimization
problem: minimizing representation entropy subject to optimal prediction
performance. Empirically, we observe an emergent memorization-compression cycle
during LLM pretraining, evidenced by oscillation positive/negative gradient
alignment between cross-entropy and Matrix-Based Entropy (MBE), a measure of
representation entropy. This pattern closely mirrors the predictive-compressive
trade-off prescribed by IBLM and also parallels the biological alternation
between awake learning and sleep consolidation. Motivated by this observation,
we propose Gated Phase Transition (GAPT), a training algorithm that adaptively
switches between memorization and compression phases. When applied to GPT-2
pretraining on FineWeb dataset, GAPT reduces MBE by 50% and improves
cross-entropy by 4.8%. GAPT improves OOD generalizatino by 35% in a pretraining
task on arithmetic multiplication. In a setting designed to simulate
catastrophic forgetting, GAPT reduces interference by compressing and
separating representations, achieving a 97% improvement in separation -
paralleling the functional role of sleep consolidation.
Information Bottleneck Language Modeling introduces a reversible memorization-compression cycle in language model training to enhance generalization and reduce overfitting.
Information Bottleneck Language Modeling IBLM constrained optimization representation entropy Matrix-Based Entropy MBE Gated Phase Transition GAPT FineWeb dataset OOD generalization catastrophic forgetting sleep consolidation