title: Scaling LLM Pre-training with Vocabulary Curriculum
summary: Modern language models rely on static vocabularies, fixed before pretraining,
  in contrast to the adaptive vocabulary acquisition observed in human language
  learning. To bridge this gap, we introduce vocabulary curriculum learning, an
  approach that improves pretraining efficiency with log-linear scaling gains
  relative to vocabulary size. Our method alternates between entropy-guided
  vocabulary expansion and model optimization, enabling models to learn
  transferable representations across diverse tokenization granularities. This
  approach naturally gives rise to an optimal computation allocation pattern:
  longer tokens capture predictable content, while shorter tokens focus on more
  complex, harder-to-predict contexts. Experiments on small-scale GPT models
  demonstrate improved scaling efficiency, reinforcing the effectiveness of
  dynamic tokenization. We release our code to support further research and plan
  to extend our experiments to larger models and diverse domains.
paper_authors._id: 67be7f96b4ca41e2807a4fb0
paper_authors.name: Fangyuan Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:26:07.778Z
paper_authors._id: 64d98ef7a4839890b25eb78b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d98ef7a4839890b25eb78b/215-CSVLl81z6CAq0ECWU.jpeg
paper_authors.fullname: Fangyuan Yu
paper_authors.type: user
paper_authors.user: Ksgk-fy
paper_title: Scaling LLM Pre-training with Vocabulary Curriculum
paper_summary: Modern language models rely on static vocabularies, fixed before pretraining,
  in contrast to the adaptive vocabulary acquisition observed in human language
  learning. To bridge this gap, we introduce vocabulary curriculum learning, an
  approach that improves pretraining efficiency with log-linear scaling gains
  relative to vocabulary size. Our method alternates between entropy-guided
  vocabulary expansion and model optimization, enabling models to learn
  transferable representations across diverse tokenization granularities. This
  approach naturally gives rise to an optimal computation allocation pattern:
  longer tokens capture predictable content, while shorter tokens focus on more
  complex, harder-to-predict contexts. Experiments on small-scale GPT models
  demonstrate improved scaling efficiency, reinforcing the effectiveness of
  dynamic tokenization. We release our code to support further research and plan
  to extend our experiments to larger models and diverse domains.
paper_ai_summary: Pioneering vocabulary curriculum learning enhances pretraining efficiency in language models by dynamically adjusting vocabulary and balancing tokenization granularity.
paper_ai_keywords: vocabulary curriculum learning entropy-guided vocabulary expansion model optimization transferable representations tokenization granularities