Reasoning to Learn from Latent Thoughts
Compute scaling for language model (LM) pretraining has outpaced the growth
of human-written texts, leading to concerns that data will become the
bottleneck to LM scaling. To continue scaling pretraining in this
data-constrained regime, we propose that explicitly modeling and inferring the
latent thoughts that underlie the text generation process can significantly
improve pretraining data efficiency. Intuitively, our approach views web text
as the compressed final outcome of a verbose human thought process and that the
latent thoughts contain important contextual knowledge and reasoning steps that
are critical to data-efficient learning. We empirically demonstrate the
effectiveness of our approach through data-constrained continued pretraining
for math. We first show that synthetic data approaches to inferring latent
thoughts significantly improve data efficiency, outperforming training on the
same amount of raw data (5.7\% rightarrow 25.4\% on MATH). Furthermore, we
demonstrate latent thought inference without a strong teacher, where an LM
bootstraps its own performance by using an EM algorithm to iteratively improve
the capability of the trained LM and the quality of thought-augmented
pretraining data. We show that a 1B LM can bootstrap its performance across at
least three iterations and significantly outperform baselines trained on raw
data, with increasing gains from additional inference compute when performing
the E-step. The gains from inference scaling and EM iterations suggest new
opportunities for scaling data-constrained pretraining.
67e2290da4525cbb1d718ae2
Yangjun Ruan
admin_assigned
2025-03-25T09:33:10.423Z
65619949d2e4352d64365606
/avatars/4241b6e901e146a9ff8b5bc7de3f960a.svg
Yangjun Ruan
user
ryoungj
67e2290da4525cbb1d718ae3
Neil Band
admin_assigned
2025-03-25T09:33:16.895Z
630bc38809eceb8fafe5ed7f
/avatars/5f2a1268f8a7b51cca8446ef0be6445f.svg
Neil Band
user
nband
67e2290da4525cbb1d718ae4
Chris J. Maddison
admin_assigned
2025-03-25T09:33:22.633Z
66a7f54fbb22d7e78a2aeaf4
/avatars/3ab8899935f8f7b14e89c623cc6c0fd2.svg
Chris J. Maddison
user
cmaddis
67e2290da4525cbb1d718ae5
Tatsunori Hashimoto
Reasoning to Learn from Latent Thoughts
Compute scaling for language model (LM) pretraining has outpaced the growth
of human-written texts, leading to concerns that data will become the
bottleneck to LM scaling. To continue scaling pretraining in this
data-constrained regime, we propose that explicitly modeling and inferring the
latent thoughts that underlie the text generation process can significantly
improve pretraining data efficiency. Intuitively, our approach views web text
as the compressed final outcome of a verbose human thought process and that the
latent thoughts contain important contextual knowledge and reasoning steps that
are critical to data-efficient learning. We empirically demonstrate the
effectiveness of our approach through data-constrained continued pretraining
for math. We first show that synthetic data approaches to inferring latent
thoughts significantly improve data efficiency, outperforming training on the
same amount of raw data (5.7\% rightarrow 25.4\% on MATH). Furthermore, we
demonstrate latent thought inference without a strong teacher, where an LM
bootstraps its own performance by using an EM algorithm to iteratively improve
the capability of the trained LM and the quality of thought-augmented
pretraining data. We show that a 1B LM can bootstrap its performance across at
least three iterations and significantly outperform baselines trained on raw
data, with increasing gains from additional inference compute when performing
the E-step. The gains from inference scaling and EM iterations suggest new
opportunities for scaling data-constrained pretraining.
Modeling and inferring latent thoughts during language model pretraining improves data efficiency and scaling potential in data-constrained regimes.
latent thoughts text generation process data-efficient learning synthetic data EM algorithm thought-augmented pretraining data inference scaling