title: LLM Pretraining with Continuous Concepts
summary: Next token prediction has been the standard training objective used in large
  language model pretraining. Representations are learned as a result of
  optimizing for token-level perplexity. We propose Continuous Concept Mixing
  (CoCoMix), a novel pretraining framework that combines discrete next token
  prediction with continuous concepts. Specifically, CoCoMix predicts continuous
  concepts learned from a pretrained sparse autoencoder and mixes them into the
  model's hidden state by interleaving with token hidden representations. Through
  experiments on multiple benchmarks, including language modeling and downstream
  reasoning tasks, we show that CoCoMix is more sample efficient and consistently
  outperforms standard next token prediction, knowledge distillation and
  inserting pause tokens. We find that combining both concept learning and
  interleaving in an end-to-end framework is critical to performance gains.
  Furthermore, CoCoMix enhances interpretability and steerability by allowing
  direct inspection and modification of the predicted concept, offering a
  transparent way to guide the model's internal reasoning process.
paper_authors._id: 67ad783da2808b57a3cd3316
paper_authors.name: Jihoon Tack
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T15:31:46.173Z
paper_authors._id: 65b15a471566a60bda376e02
paper_authors.avatarUrl: /avatars/413340cb2d44c765accb5b3fc450ccc4.svg
paper_authors.fullname: Jihoon Tack
paper_authors.type: user
paper_authors.user: jihoontack
paper_authors._id: 67ad783da2808b57a3cd3317
paper_authors.name: Jack Lanchantin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T15:31:36.784Z
paper_authors._id: 65dd30a6ea1a202d3cb469f7
paper_authors.avatarUrl: /avatars/14029def026380e6e20822d916b25a72.svg
paper_authors.fullname: Jack Lanchantin
paper_authors.type: user
paper_authors.user: jcklcn
paper_authors._id: 67ad783da2808b57a3cd3318
paper_authors.name: Jane Yu
paper_authors._id: 67ad783da2808b57a3cd3319
paper_authors.name: Andrew Cohen
paper_authors._id: 67ad783da2808b57a3cd331a
paper_authors.name: Ilia Kulikov
paper_authors._id: 67ad783da2808b57a3cd331b
paper_authors.name: Janice Lan
paper_authors._id: 67ad783da2808b57a3cd331c
paper_authors.name: Shibo Hao
paper_authors._id: 67ad783da2808b57a3cd331d
paper_authors.name: Yuandong Tian
paper_authors._id: 67ad783da2808b57a3cd331e
paper_authors.name: Jason Weston
paper_authors._id: 67ad783da2808b57a3cd331f
paper_authors.name: Xian Li
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-13T04:42:38.302Z
paper_authors._id: 659a395421a7431643caedda
paper_authors.avatarUrl: /avatars/c1e0bbcedce68fe3b4fe39e0cf01c65c.svg
paper_authors.fullname: Xian Li
paper_authors.type: user
paper_authors.user: xlxxl
paper_title: LLM Pretraining with Continuous Concepts
paper_summary: Next token prediction has been the standard training objective used in large
  language model pretraining. Representations are learned as a result of
  optimizing for token-level perplexity. We propose Continuous Concept Mixing
  (CoCoMix), a novel pretraining framework that combines discrete next token
  prediction with continuous concepts. Specifically, CoCoMix predicts continuous
  concepts learned from a pretrained sparse autoencoder and mixes them into the
  model's hidden state by interleaving with token hidden representations. Through
  experiments on multiple benchmarks, including language modeling and downstream
  reasoning tasks, we show that CoCoMix is more sample efficient and consistently
  outperforms standard next token prediction, knowledge distillation and
  inserting pause tokens. We find that combining both concept learning and
  interleaving in an end-to-end framework is critical to performance gains.
  Furthermore, CoCoMix enhances interpretability and steerability by allowing
  direct inspection and modification of the predicted concept, offering a
  transparent way to guide the model's internal reasoning process.
paper_ai_keywords: Continuous Concept Mixing (CoCoMix) next token prediction token-level perplexity sparse autoencoder continuous concepts hidden state token hidden representations language modeling downstream reasoning tasks sample efficient performance gains interpretability steerability predicted concept internal reasoning process