Let's Predict Sentence by Sentence
Autoregressive language models (LMs) generate one token at a time, yet human
reasoning operates over higher-level abstractions - sentences, propositions,
and concepts. This contrast raises a central question- Can LMs likewise learn
to reason over structured semantic units rather than raw token sequences? In
this work, we investigate whether pretrained LMs can be lifted into such
abstract reasoning spaces by building on their learned representations. We
present a framework that adapts a pretrained token-level LM to operate in
sentence space by autoregressively predicting continuous embeddings of next
sentences. We explore two embedding paradigms inspired by classical
representation learning: 1) semantic embeddings, learned via autoencoding to
preserve surface meaning; and 2) contextual embeddings, trained via
next-sentence prediction to encode anticipatory structure. We evaluate both
under two inference regimes: Discretized, which decodes each predicted
embedding into text before re-encoding; and Continuous, which reasons entirely
in embedding space for improved efficiency. Across four domains - mathematics,
logic, commonsense, and planning - contextual embeddings under continuous
inference show competitive performance with Chain-of-Thought (CoT) while
reducing inference-time FLOPs on average by half. We also present early signs
of scalability and modular adaptation. Finally, to visualize latent
trajectories, we introduce SentenceLens, a diagnostic tool that decodes
intermediate model states into interpretable sentences. Together, our results
indicate that pretrained LMs can effectively transition to abstract, structured
reasoning within latent embedding spaces.
6837eff312d1f7a138bd09b3
Hyeonbin Hwang
claimed_verified
2025-05-30T06:54:51.285Z
647eaaf61a1fcad2fdc5d1ef
/avatars/596d7a61d6e6227d38b661210a32fed0.svg
Hyeonbin Hwang 
user
hbin0701
6837eff312d1f7a138bd09b4
Byeongguk Jeon
6837eff312d1f7a138bd09b5
Seungone Kim
claimed_verified
2025-05-29T20:19:02.694Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
user
seungone
6837eff312d1f7a138bd09b6
Jiyeon Kim
6837eff312d1f7a138bd09b7
Hoyeon Chang
6837eff312d1f7a138bd09b8
Sohee Yang
6837eff312d1f7a138bd09b9
Seungpil Won
6837eff312d1f7a138bd09ba
Dohaeng Lee
6837eff312d1f7a138bd09bb
Youbin Ahn
6837eff312d1f7a138bd09bc
Minjoon Seo
Let's Predict Sentence by Sentence
Autoregressive language models (LMs) generate one token at a time, yet human
reasoning operates over higher-level abstractions - sentences, propositions,
and concepts. This contrast raises a central question- Can LMs likewise learn
to reason over structured semantic units rather than raw token sequences? In
this work, we investigate whether pretrained LMs can be lifted into such
abstract reasoning spaces by building on their learned representations. We
present a framework that adapts a pretrained token-level LM to operate in
sentence space by autoregressively predicting continuous embeddings of next
sentences. We explore two embedding paradigms inspired by classical
representation learning: 1) semantic embeddings, learned via autoencoding to
preserve surface meaning; and 2) contextual embeddings, trained via
next-sentence prediction to encode anticipatory structure. We evaluate both
under two inference regimes: Discretized, which decodes each predicted
embedding into text before re-encoding; and Continuous, which reasons entirely
in embedding space for improved efficiency. Across four domains - mathematics,
logic, commonsense, and planning - contextual embeddings under continuous
inference show competitive performance with Chain-of-Thought (CoT) while
reducing inference-time FLOPs on average by half. We also present early signs
of scalability and modular adaptation. Finally, to visualize latent
trajectories, we introduce SentenceLens, a diagnostic tool that decodes
intermediate model states into interpretable sentences. Together, our results
indicate that pretrained LMs can effectively transition to abstract, structured
reasoning within latent embedding spaces.
Pretrained language models can adapt to operate in sentence space, reason over structured semantic units, and achieve competitive performance with Chain-of-Thought while reducing inference-time FLOPs.
autoregressive language models semantic embeddings contextual embeddings next-sentence prediction Chain-of-Thought SentenceLens