title: Hybrid Latent Reasoning via Reinforcement Learning
summary: Recent advances in large language models (LLMs) have introduced latent
  reasoning as a promising alternative to autoregressive reasoning. By performing
  internal computation with hidden states from previous steps, latent reasoning
  benefit from more informative features rather than sampling a discrete
  chain-of-thought (CoT) path. Yet latent reasoning approaches are often
  incompatible with LLMs, as their continuous paradigm conflicts with the
  discrete nature of autoregressive generation. Moreover, these methods rely on
  CoT traces for training and thus fail to exploit the inherent reasoning
  patterns of LLMs. In this work, we explore latent reasoning by leveraging the
  intrinsic capabilities of LLMs via reinforcement learning (RL). To this end, we
  introduce hybrid reasoning policy optimization (HRPO), an RL-based hybrid
  latent reasoning approach that (1) integrates prior hidden states into sampled
  tokens with a learnable gating mechanism, and (2) initializes training with
  predominantly token embeddings while progressively incorporating more hidden
  features. This design maintains LLMs' generative capabilities and incentivizes
  hybrid reasoning using both discrete and continuous representations. In
  addition, the hybrid HRPO introduces stochasticity into latent reasoning via
  token sampling, thereby enabling RL-based optimization without requiring CoT
  trajectories. Extensive evaluations across diverse benchmarks show that HRPO
  outperforms prior methods in both knowledge- and reasoning-intensive tasks.
  Furthermore, HRPO-trained LLMs remain interpretable and exhibit intriguing
  behaviors like cross-lingual patterns and shorter completion lengths,
  highlighting the potential of our RL-based approach and offer insights for
  future work in latent reasoning.
paper_authors._id: 683606b33e467355094d95bc
paper_authors.name: Zhenrui Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:58:55.873Z
paper_authors._id: 644c4eff9a7c19cfaf0c8f4a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/2gHtfhPFjSsnwxVtBnxe2.jpeg
paper_authors.fullname: Zhenrui Yue
paper_authors.type: user
paper_authors.user: yuezrhb
paper_authors._id: 683606b33e467355094d95bd
paper_authors.name: Bowen Jin
paper_authors._id: 683606b33e467355094d95be
paper_authors.name: Huimin Zeng
paper_authors._id: 683606b33e467355094d95bf
paper_authors.name: Honglei Zhuang
paper_authors._id: 683606b33e467355094d95c0
paper_authors.name: Zhen Qin
paper_authors._id: 683606b33e467355094d95c1
paper_authors.name: Jinsung Yoon
paper_authors._id: 683606b33e467355094d95c2
paper_authors.name: Lanyu Shang
paper_authors._id: 683606b33e467355094d95c3
paper_authors.name: Jiawei Han
paper_authors._id: 683606b33e467355094d95c4
paper_authors.name: Dong Wang
paper_title: Hybrid Latent Reasoning via Reinforcement Learning
paper_summary: Recent advances in large language models (LLMs) have introduced latent
  reasoning as a promising alternative to autoregressive reasoning. By performing
  internal computation with hidden states from previous steps, latent reasoning
  benefit from more informative features rather than sampling a discrete
  chain-of-thought (CoT) path. Yet latent reasoning approaches are often
  incompatible with LLMs, as their continuous paradigm conflicts with the
  discrete nature of autoregressive generation. Moreover, these methods rely on
  CoT traces for training and thus fail to exploit the inherent reasoning
  patterns of LLMs. In this work, we explore latent reasoning by leveraging the
  intrinsic capabilities of LLMs via reinforcement learning (RL). To this end, we
  introduce hybrid reasoning policy optimization (HRPO), an RL-based hybrid
  latent reasoning approach that (1) integrates prior hidden states into sampled
  tokens with a learnable gating mechanism, and (2) initializes training with
  predominantly token embeddings while progressively incorporating more hidden
  features. This design maintains LLMs' generative capabilities and incentivizes
  hybrid reasoning using both discrete and continuous representations. In
  addition, the hybrid HRPO introduces stochasticity into latent reasoning via
  token sampling, thereby enabling RL-based optimization without requiring CoT
  trajectories. Extensive evaluations across diverse benchmarks show that HRPO
  outperforms prior methods in both knowledge- and reasoning-intensive tasks.
  Furthermore, HRPO-trained LLMs remain interpretable and exhibit intriguing
  behaviors like cross-lingual patterns and shorter completion lengths,
  highlighting the potential of our RL-based approach and offer insights for
  future work in latent reasoning.
paper_ai_summary: Hybrid reasoning policy optimization (HRPO) leverages reinforcement learning to integrate latent reasoning with large language models, enhancing performance in knowledge- and reasoning-intensive tasks while maintaining interpretability.
paper_ai_keywords: latent reasoning autoregressive reasoning hidden states chain-of-thought (CoT) reinforcement learning (RL) hybrid reasoning HRPO token embeddings generative capabilities discrete representations continuous representations stochasticity token sampling cross-lingual patterns completion lengths