title: Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
summary: Research on scaling large language models (LLMs) has primarily focused on
  model parameters and training data size, overlooking the role of vocabulary
  size. % Intuitively, larger vocabularies enable more efficient tokenization by
  representing sentences with fewer tokens, but they also increase the risk of
  under-fitting representations for rare tokens. We investigate how vocabulary
  size impacts LLM scaling laws by training models ranging from 33M to 3B
  parameters on up to 500B characters with various vocabulary configurations. We
  propose three complementary approaches for predicting the compute-optimal
  vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit
  of the loss function. Our approaches converge on the same result that the
  optimal vocabulary size depends on the available compute budget and that larger
  models deserve larger vocabularies. However, most LLMs use too small vocabulary
  sizes. For example, we predict that the optimal vocabulary size of Llama2-70B
  should have been at least 216K, 7 times larger than its vocabulary of 32K. We
  validate our predictions empirically by training models with 3B parameters
  across different FLOPs budgets. Adopting our predicted optimal vocabulary size
  consistently improves downstream performance over commonly used vocabulary
  sizes. By increasing the vocabulary size from the conventional 32K to 43K, we
  improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21
  FLOPs. Our work emphasizes the necessity of jointly considering model
  parameters and vocabulary size for efficient scaling.
paper_authors._id: 6699cc6534b724c13d591358
paper_authors.name: Chaofan Tao
paper_authors._id: 6699cc6534b724c13d591359
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:23:17.971Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 6699cc6534b724c13d59135a
paper_authors.name: Longxu Dou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:23:15.966Z
paper_authors._id: 6214e4ee1e35c843d42d1f88
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6214e4ee1e35c843d42d1f88/fj-9wuIdPhvogh3BrcXTB.jpeg
paper_authors.fullname: Longxu Dou
paper_authors.type: user
paper_authors.user: dreamerdeo
paper_authors._id: 6699cc6534b724c13d59135b
paper_authors.name: Niklas Muennighoff
paper_authors._id: 6699cc6534b724c13d59135c
paper_authors.name: Zhongwei Wan
paper_authors._id: 6699cc6534b724c13d59135d
paper_authors.name: Ping Luo
paper_authors._id: 6699cc6534b724c13d59135e
paper_authors.name: Min Lin
paper_authors._id: 6699cc6534b724c13d59135f
paper_authors.name: Ngai Wong
paper_title: Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
paper_summary: Research on scaling large language models (LLMs) has primarily focused on
  model parameters and training data size, overlooking the role of vocabulary
  size. % Intuitively, larger vocabularies enable more efficient tokenization by
  representing sentences with fewer tokens, but they also increase the risk of
  under-fitting representations for rare tokens. We investigate how vocabulary
  size impacts LLM scaling laws by training models ranging from 33M to 3B
  parameters on up to 500B characters with various vocabulary configurations. We
  propose three complementary approaches for predicting the compute-optimal
  vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit
  of the loss function. Our approaches converge on the same result that the
  optimal vocabulary size depends on the available compute budget and that larger
  models deserve larger vocabularies. However, most LLMs use too small vocabulary
  sizes. For example, we predict that the optimal vocabulary size of Llama2-70B
  should have been at least 216K, 7 times larger than its vocabulary of 32K. We
  validate our predictions empirically by training models with 3B parameters
  across different FLOPs budgets. Adopting our predicted optimal vocabulary size
  consistently improves downstream performance over commonly used vocabulary
  sizes. By increasing the vocabulary size from the conventional 32K to 43K, we
  improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21
  FLOPs. Our work emphasizes the necessity of jointly considering model
  parameters and vocabulary size for efficient scaling.
paper_ai_keywords: neural language models (LLMs) tokenization zero-fitting representations scaling laws IsoFLOPs analysis Evaluative parametric fit loss function FLOPs budgets ARC-Challenge vocabulary size conventional vocabulary size downstream performance