title: Let's Predict Sentence by Sentence
summary: Autoregressive language models (LMs) generate one token at a time, yet human
  reasoning operates over higher-level abstractions - sentences, propositions,
  and concepts. This contrast raises a central question- Can LMs likewise learn
  to reason over structured semantic units rather than raw token sequences? In
  this work, we investigate whether pretrained LMs can be lifted into such
  abstract reasoning spaces by building on their learned representations. We
  present a framework that adapts a pretrained token-level LM to operate in
  sentence space by autoregressively predicting continuous embeddings of next
  sentences. We explore two embedding paradigms inspired by classical
  representation learning: 1) semantic embeddings, learned via autoencoding to
  preserve surface meaning; and 2) contextual embeddings, trained via
  next-sentence prediction to encode anticipatory structure. We evaluate both
  under two inference regimes: Discretized, which decodes each predicted
  embedding into text before re-encoding; and Continuous, which reasons entirely
  in embedding space for improved efficiency. Across four domains - mathematics,
  logic, commonsense, and planning - contextual embeddings under continuous
  inference show competitive performance with Chain-of-Thought (CoT) while
  reducing inference-time FLOPs on average by half. We also present early signs
  of scalability and modular adaptation. Finally, to visualize latent
  trajectories, we introduce SentenceLens, a diagnostic tool that decodes
  intermediate model states into interpretable sentences. Together, our results
  indicate that pretrained LMs can effectively transition to abstract, structured
  reasoning within latent embedding spaces.
paper_authors._id: 6837eff312d1f7a138bd09b3
paper_authors.name: Hyeonbin Hwang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:51.285Z
paper_authors._id: 647eaaf61a1fcad2fdc5d1ef
paper_authors.avatarUrl: /avatars/596d7a61d6e6227d38b661210a32fed0.svg
paper_authors.fullname: Hyeonbin Hwang 
paper_authors.type: user
paper_authors.user: hbin0701
paper_authors._id: 6837eff312d1f7a138bd09b4
paper_authors.name: Byeongguk Jeon
paper_authors._id: 6837eff312d1f7a138bd09b5
paper_authors.name: Seungone Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T20:19:02.694Z
paper_authors._id: 6469949654873f0043b09c22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
paper_authors.fullname: Seungone Kim
paper_authors.type: user
paper_authors.user: seungone
paper_authors._id: 6837eff312d1f7a138bd09b6
paper_authors.name: Jiyeon Kim
paper_authors._id: 6837eff312d1f7a138bd09b7
paper_authors.name: Hoyeon Chang
paper_authors._id: 6837eff312d1f7a138bd09b8
paper_authors.name: Sohee Yang
paper_authors._id: 6837eff312d1f7a138bd09b9
paper_authors.name: Seungpil Won
paper_authors._id: 6837eff312d1f7a138bd09ba
paper_authors.name: Dohaeng Lee
paper_authors._id: 6837eff312d1f7a138bd09bb
paper_authors.name: Youbin Ahn
paper_authors._id: 6837eff312d1f7a138bd09bc
paper_authors.name: Minjoon Seo
paper_title: Let's Predict Sentence by Sentence
paper_summary: Autoregressive language models (LMs) generate one token at a time, yet human
  reasoning operates over higher-level abstractions - sentences, propositions,
  and concepts. This contrast raises a central question- Can LMs likewise learn
  to reason over structured semantic units rather than raw token sequences? In
  this work, we investigate whether pretrained LMs can be lifted into such
  abstract reasoning spaces by building on their learned representations. We
  present a framework that adapts a pretrained token-level LM to operate in
  sentence space by autoregressively predicting continuous embeddings of next
  sentences. We explore two embedding paradigms inspired by classical
  representation learning: 1) semantic embeddings, learned via autoencoding to
  preserve surface meaning; and 2) contextual embeddings, trained via
  next-sentence prediction to encode anticipatory structure. We evaluate both
  under two inference regimes: Discretized, which decodes each predicted
  embedding into text before re-encoding; and Continuous, which reasons entirely
  in embedding space for improved efficiency. Across four domains - mathematics,
  logic, commonsense, and planning - contextual embeddings under continuous
  inference show competitive performance with Chain-of-Thought (CoT) while
  reducing inference-time FLOPs on average by half. We also present early signs
  of scalability and modular adaptation. Finally, to visualize latent
  trajectories, we introduce SentenceLens, a diagnostic tool that decodes
  intermediate model states into interpretable sentences. Together, our results
  indicate that pretrained LMs can effectively transition to abstract, structured
  reasoning within latent embedding spaces.
paper_ai_summary: Pretrained language models can adapt to operate in sentence space, reason over structured semantic units, and achieve competitive performance with Chain-of-Thought while reducing inference-time FLOPs.
paper_ai_keywords: autoregressive language models semantic embeddings contextual embeddings next-sentence prediction Chain-of-Thought SentenceLens