title: Reasoning to Learn from Latent Thoughts
summary: Compute scaling for language model (LM) pretraining has outpaced the growth
  of human-written texts, leading to concerns that data will become the
  bottleneck to LM scaling. To continue scaling pretraining in this
  data-constrained regime, we propose that explicitly modeling and inferring the
  latent thoughts that underlie the text generation process can significantly
  improve pretraining data efficiency. Intuitively, our approach views web text
  as the compressed final outcome of a verbose human thought process and that the
  latent thoughts contain important contextual knowledge and reasoning steps that
  are critical to data-efficient learning. We empirically demonstrate the
  effectiveness of our approach through data-constrained continued pretraining
  for math. We first show that synthetic data approaches to inferring latent
  thoughts significantly improve data efficiency, outperforming training on the
  same amount of raw data (5.7\% rightarrow 25.4\% on MATH). Furthermore, we
  demonstrate latent thought inference without a strong teacher, where an LM
  bootstraps its own performance by using an EM algorithm to iteratively improve
  the capability of the trained LM and the quality of thought-augmented
  pretraining data. We show that a 1B LM can bootstrap its performance across at
  least three iterations and significantly outperform baselines trained on raw
  data, with increasing gains from additional inference compute when performing
  the E-step. The gains from inference scaling and EM iterations suggest new
  opportunities for scaling data-constrained pretraining.
paper_authors._id: 67e2290da4525cbb1d718ae2
paper_authors.name: Yangjun Ruan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:33:10.423Z
paper_authors._id: 65619949d2e4352d64365606
paper_authors.avatarUrl: /avatars/4241b6e901e146a9ff8b5bc7de3f960a.svg
paper_authors.fullname: Yangjun Ruan
paper_authors.type: user
paper_authors.user: ryoungj
paper_authors._id: 67e2290da4525cbb1d718ae3
paper_authors.name: Neil Band
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:33:16.895Z
paper_authors._id: 630bc38809eceb8fafe5ed7f
paper_authors.avatarUrl: /avatars/5f2a1268f8a7b51cca8446ef0be6445f.svg
paper_authors.fullname: Neil Band
paper_authors.type: user
paper_authors.user: nband
paper_authors._id: 67e2290da4525cbb1d718ae4
paper_authors.name: Chris J. Maddison
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:33:22.633Z
paper_authors._id: 66a7f54fbb22d7e78a2aeaf4
paper_authors.avatarUrl: /avatars/3ab8899935f8f7b14e89c623cc6c0fd2.svg
paper_authors.fullname: Chris J. Maddison
paper_authors.type: user
paper_authors.user: cmaddis
paper_authors._id: 67e2290da4525cbb1d718ae5
paper_authors.name: Tatsunori Hashimoto
paper_title: Reasoning to Learn from Latent Thoughts
paper_summary: Compute scaling for language model (LM) pretraining has outpaced the growth
  of human-written texts, leading to concerns that data will become the
  bottleneck to LM scaling. To continue scaling pretraining in this
  data-constrained regime, we propose that explicitly modeling and inferring the
  latent thoughts that underlie the text generation process can significantly
  improve pretraining data efficiency. Intuitively, our approach views web text
  as the compressed final outcome of a verbose human thought process and that the
  latent thoughts contain important contextual knowledge and reasoning steps that
  are critical to data-efficient learning. We empirically demonstrate the
  effectiveness of our approach through data-constrained continued pretraining
  for math. We first show that synthetic data approaches to inferring latent
  thoughts significantly improve data efficiency, outperforming training on the
  same amount of raw data (5.7\% rightarrow 25.4\% on MATH). Furthermore, we
  demonstrate latent thought inference without a strong teacher, where an LM
  bootstraps its own performance by using an EM algorithm to iteratively improve
  the capability of the trained LM and the quality of thought-augmented
  pretraining data. We show that a 1B LM can bootstrap its performance across at
  least three iterations and significantly outperform baselines trained on raw
  data, with increasing gains from additional inference compute when performing
  the E-step. The gains from inference scaling and EM iterations suggest new
  opportunities for scaling data-constrained pretraining.
paper_ai_summary: Modeling and inferring latent thoughts during language model pretraining improves data efficiency and scaling potential in data-constrained regimes.
paper_ai_keywords: latent thoughts text generation process data-efficient learning synthetic data EM algorithm thought-augmented pretraining data inference scaling