title: Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided
    Language Modeling
summary: Using large teacher models to guide the training of smaller student models
  has become the prevailing paradigm for efficient and effective learning.
  However, vocabulary mismatches between teacher and student language models pose
  significant challenges in language modeling, resulting in divergent token
  sequences and output distributions. To overcome these limitations, we propose
  Vocabulary-agnostic Teacher Guided Language Modeling (VocAgnoLM), a novel
  approach that bridges the gap caused by vocabulary mismatch through two key
  methods: (1) Token-level Lexical Alignment, which aligns token sequences across
  mismatched vocabularies, and (2) Teacher Guided Loss, which leverages the loss
  of teacher model to guide effective student training. We demonstrate its
  effectiveness in language modeling with 1B student model using various 7B
  teacher models with different vocabularies. Notably, with
  Qwen2.5-Math-Instruct, a teacher model sharing only about 6% of its vocabulary
  with TinyLlama, VocAgnoLM achieves a 46% performance improvement compared to
  naive continual pretraining. Furthermore, we demonstrate that VocAgnoLM
  consistently benefits from stronger teacher models, providing a robust solution
  to vocabulary mismatches in language modeling.
paper_authors._id: 67e3fcb76323be71222fce35
paper_authors.name: Haebin Shin
paper_authors._id: 67e3fcb76323be71222fce36
paper_authors.name: Lei Ji
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-26T13:10:17.004Z
paper_authors._id: 664bed934bea570e25a8dc8c
paper_authors.avatarUrl: /avatars/4e5a69db9005759b8812a9b54faf14c1.svg
paper_authors.fullname: Lei Ji
paper_authors.type: user
paper_authors.user: jilei03111
paper_authors._id: 67e3fcb76323be71222fce37
paper_authors.name: Xiao Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-26T13:10:33.192Z
paper_authors._id: 63fb6e281b4b1bd4e7ffc5be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: lx865712528
paper_authors._id: 67e3fcb76323be71222fce38
paper_authors.name: Yeyun Gong
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-26T13:10:17.004Z
paper_authors._id: 643f615aa16cd6d1f4c581de
paper_authors.avatarUrl: /avatars/47753a3e82b44f81881600c52e1e8495.svg
paper_authors.fullname: Yeyun Gong
paper_authors.type: user
paper_authors.user: yegong
paper_title: Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided
    Language Modeling
paper_summary: Using large teacher models to guide the training of smaller student models
  has become the prevailing paradigm for efficient and effective learning.
  However, vocabulary mismatches between teacher and student language models pose
  significant challenges in language modeling, resulting in divergent token
  sequences and output distributions. To overcome these limitations, we propose
  Vocabulary-agnostic Teacher Guided Language Modeling (VocAgnoLM), a novel
  approach that bridges the gap caused by vocabulary mismatch through two key
  methods: (1) Token-level Lexical Alignment, which aligns token sequences across
  mismatched vocabularies, and (2) Teacher Guided Loss, which leverages the loss
  of teacher model to guide effective student training. We demonstrate its
  effectiveness in language modeling with 1B student model using various 7B
  teacher models with different vocabularies. Notably, with
  Qwen2.5-Math-Instruct, a teacher model sharing only about 6% of its vocabulary
  with TinyLlama, VocAgnoLM achieves a 46% performance improvement compared to
  naive continual pretraining. Furthermore, we demonstrate that VocAgnoLM
  consistently benefits from stronger teacher models, providing a robust solution
  to vocabulary mismatches in language modeling.
paper_ai_summary: VocAgnoLM addresses vocabulary mismatches between teacher and student models through token-level alignment and teacher-guided loss, improving performance in language modeling.
paper_ai_keywords: Token-level Lexical Alignment Teacher Guided Loss VocAgnoLM language modeling student model teacher model vocabulary mismatches Qwen2.5-Math-Instruct TinyLlama