title: Multi-Token Prediction Needs Registers
summary: Multi-token prediction has emerged as a promising objective for improving
  language model pretraining, but its benefits have not consistently generalized
  to other settings such as fine-tuning. In this paper, we propose MuToR, a
  simple and effective approach to multi-token prediction that interleaves
  learnable register tokens into the input sequence, each tasked with predicting
  future targets. Compared to existing methods, MuToR offers several key
  advantages: it introduces only a negligible number of additional parameters,
  requires no architectural changes--ensuring compatibility with off-the-shelf
  pretrained language models--and remains aligned with the next-token pretraining
  objective, making it especially well-suited for supervised fine-tuning.
  Moreover, it naturally supports scalable prediction horizons. We demonstrate
  the effectiveness and versatility of MuToR across a range of use cases,
  including supervised fine-tuning, parameter-efficient fine-tuning (PEFT), and
  pretraining, on challenging generative tasks in both language and vision
  domains. Our code will be available at: https://github.com/nasosger/MuToR.
paper_authors._id: 68271c682f2e31ef0667bfaf
paper_authors.name: Anastasios Gerontopoulos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-18T19:39:25.735Z
paper_authors._id: 668e4d1b446c8736208d99e1
paper_authors.avatarUrl: /avatars/dbe10f3b181e789d98b9b6bde4f711b2.svg
paper_authors.fullname: Anastasios Gerontopoulos
paper_authors.type: user
paper_authors.user: nasos10
paper_authors._id: 68271c682f2e31ef0667bfb0
paper_authors.name: Spyros Gidaris
paper_authors._id: 68271c682f2e31ef0667bfb1
paper_authors.name: Nikos Komodakis
paper_title: Multi-Token Prediction Needs Registers
paper_summary: Multi-token prediction has emerged as a promising objective for improving
  language model pretraining, but its benefits have not consistently generalized
  to other settings such as fine-tuning. In this paper, we propose MuToR, a
  simple and effective approach to multi-token prediction that interleaves
  learnable register tokens into the input sequence, each tasked with predicting
  future targets. Compared to existing methods, MuToR offers several key
  advantages: it introduces only a negligible number of additional parameters,
  requires no architectural changes--ensuring compatibility with off-the-shelf
  pretrained language models--and remains aligned with the next-token pretraining
  objective, making it especially well-suited for supervised fine-tuning.
  Moreover, it naturally supports scalable prediction horizons. We demonstrate
  the effectiveness and versatility of MuToR across a range of use cases,
  including supervised fine-tuning, parameter-efficient fine-tuning (PEFT), and
  pretraining, on challenging generative tasks in both language and vision
  domains. Our code will be available at: https://github.com/nasosger/MuToR.
paper_ai_summary: MuToR is a multi-token prediction approach that integrates learnable register tokens to future target prediction, enhancing language model pretraining and fine-tuning without significant parameter increase or architectural changes.
paper_ai_keywords: multi-token prediction learnable register tokens fine-tuning parameter-efficient fine-tuning pretraining next-token pretraining objective generative tasks