title: Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
summary: Tokenization is a fundamental component of large language models (LLMs), yet
  its influence on model scaling and performance is not fully explored. In this
  paper, we introduce Over-Tokenized Transformers, a novel framework that
  decouples input and output vocabularies to improve language modeling
  performance. Specifically, our approach scales up input vocabularies to
  leverage multi-gram tokens. Through extensive experiments, we uncover a
  log-linear relationship between input vocabulary size and training loss,
  demonstrating that larger input vocabularies consistently enhance model
  performance, regardless of model size. Using a large input vocabulary, we
  achieve performance comparable to double-sized baselines with no additional
  cost. Our findings highlight the importance of tokenization in scaling laws and
  provide practical insight for tokenizer design, paving the way for more
  efficient and powerful LLMs.
paper_authors._id: 6799b345a66ae6b357bef986
paper_authors.name: Hongzhi Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T17:16:08.642Z
paper_authors._id: 66fbbed8a0154e0b6498293d
paper_authors.avatarUrl: /avatars/5c4202f13cc5af6424154fae293fad52.svg
paper_authors.fullname: Huang Hongzhi
paper_authors.type: user
paper_authors.user: xyzed
paper_authors._id: 6799b345a66ae6b357bef987
paper_authors.name: Defa Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T17:16:06.916Z
paper_authors._id: 667505f4361b960c79e35486
paper_authors.avatarUrl: /avatars/d352639c520075220f6abaae23c39376.svg
paper_authors.fullname: Defa Zhu
paper_authors.type: user
paper_authors.user: mathfinder
paper_authors._id: 6799b345a66ae6b357bef988
paper_authors.name: Banggu Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:21:12.737Z
paper_authors._id: 64a3b0fa565496b629879293
paper_authors.avatarUrl: /avatars/9d5de6cc4a01052ea971701f72bd3489.svg
paper_authors.fullname: wubanggu
paper_authors.type: user
paper_authors.user: banggu
paper_authors._id: 6799b345a66ae6b357bef989
paper_authors.name: Yutao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T17:16:10.495Z
paper_authors._id: 6371128eafbe42caa5a5222b
paper_authors.avatarUrl: /avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
paper_authors.fullname: Yutao Zeng
paper_authors.type: user
paper_authors.user: Taoer
paper_authors._id: 6799b345a66ae6b357bef98a
paper_authors.name: Ya Wang
paper_authors._id: 6799b345a66ae6b357bef98b
paper_authors.name: Qiyang Min
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:21:23.469Z
paper_authors._id: 645507b89d37c3fb33279fe3
paper_authors.avatarUrl: /avatars/ec1122137c49204ab968182d1f726c35.svg
paper_authors.fullname: min
paper_authors.type: user
paper_authors.user: qiyang-attn
paper_authors._id: 6799b345a66ae6b357bef98c
paper_authors.name: Xun Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:21:37.188Z
paper_authors._id: 62533db4a06ec75172eeabe7
paper_authors.avatarUrl: /avatars/b1a4dad90afae5c00df97233a97777db.svg
paper_authors.fullname: xunzhou
paper_authors.type: user
paper_authors.user: xunzhou
paper_title: Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
paper_summary: Tokenization is a fundamental component of large language models (LLMs), yet
  its influence on model scaling and performance is not fully explored. In this
  paper, we introduce Over-Tokenized Transformers, a novel framework that
  decouples input and output vocabularies to improve language modeling
  performance. Specifically, our approach scales up input vocabularies to
  leverage multi-gram tokens. Through extensive experiments, we uncover a
  log-linear relationship between input vocabulary size and training loss,
  demonstrating that larger input vocabularies consistently enhance model
  performance, regardless of model size. Using a large input vocabulary, we
  achieve performance comparable to double-sized baselines with no additional
  cost. Our findings highlight the importance of tokenization in scaling laws and
  provide practical insight for tokenizer design, paving the way for more
  efficient and powerful LLMs.
paper_ai_keywords: Over-Tokenized Transformers multi-gram tokens input vocabulary size training loss scaling laws tokenizer design