Hybrid Latent Reasoning via Reinforcement Learning
Recent advances in large language models (LLMs) have introduced latent
reasoning as a promising alternative to autoregressive reasoning. By performing
internal computation with hidden states from previous steps, latent reasoning
benefit from more informative features rather than sampling a discrete
chain-of-thought (CoT) path. Yet latent reasoning approaches are often
incompatible with LLMs, as their continuous paradigm conflicts with the
discrete nature of autoregressive generation. Moreover, these methods rely on
CoT traces for training and thus fail to exploit the inherent reasoning
patterns of LLMs. In this work, we explore latent reasoning by leveraging the
intrinsic capabilities of LLMs via reinforcement learning (RL). To this end, we
introduce hybrid reasoning policy optimization (HRPO), an RL-based hybrid
latent reasoning approach that (1) integrates prior hidden states into sampled
tokens with a learnable gating mechanism, and (2) initializes training with
predominantly token embeddings while progressively incorporating more hidden
features. This design maintains LLMs' generative capabilities and incentivizes
hybrid reasoning using both discrete and continuous representations. In
addition, the hybrid HRPO introduces stochasticity into latent reasoning via
token sampling, thereby enabling RL-based optimization without requiring CoT
trajectories. Extensive evaluations across diverse benchmarks show that HRPO
outperforms prior methods in both knowledge- and reasoning-intensive tasks.
Furthermore, HRPO-trained LLMs remain interpretable and exhibit intriguing
behaviors like cross-lingual patterns and shorter completion lengths,
highlighting the potential of our RL-based approach and offer insights for
future work in latent reasoning.
683606b33e467355094d95bc
Zhenrui Yue
claimed_verified
2025-05-28T08:58:55.873Z
644c4eff9a7c19cfaf0c8f4a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/2gHtfhPFjSsnwxVtBnxe2.jpeg
Zhenrui Yue
user
yuezrhb
683606b33e467355094d95bd
Bowen Jin
683606b33e467355094d95be
Huimin Zeng
683606b33e467355094d95bf
Honglei Zhuang
683606b33e467355094d95c0
Zhen Qin
683606b33e467355094d95c1
Jinsung Yoon
683606b33e467355094d95c2
Lanyu Shang
683606b33e467355094d95c3
Jiawei Han
683606b33e467355094d95c4
Dong Wang
Hybrid Latent Reasoning via Reinforcement Learning
Recent advances in large language models (LLMs) have introduced latent
reasoning as a promising alternative to autoregressive reasoning. By performing
internal computation with hidden states from previous steps, latent reasoning
benefit from more informative features rather than sampling a discrete
chain-of-thought (CoT) path. Yet latent reasoning approaches are often
incompatible with LLMs, as their continuous paradigm conflicts with the
discrete nature of autoregressive generation. Moreover, these methods rely on
CoT traces for training and thus fail to exploit the inherent reasoning
patterns of LLMs. In this work, we explore latent reasoning by leveraging the
intrinsic capabilities of LLMs via reinforcement learning (RL). To this end, we
introduce hybrid reasoning policy optimization (HRPO), an RL-based hybrid
latent reasoning approach that (1) integrates prior hidden states into sampled
tokens with a learnable gating mechanism, and (2) initializes training with
predominantly token embeddings while progressively incorporating more hidden
features. This design maintains LLMs' generative capabilities and incentivizes
hybrid reasoning using both discrete and continuous representations. In
addition, the hybrid HRPO introduces stochasticity into latent reasoning via
token sampling, thereby enabling RL-based optimization without requiring CoT
trajectories. Extensive evaluations across diverse benchmarks show that HRPO
outperforms prior methods in both knowledge- and reasoning-intensive tasks.
Furthermore, HRPO-trained LLMs remain interpretable and exhibit intriguing
behaviors like cross-lingual patterns and shorter completion lengths,
highlighting the potential of our RL-based approach and offer insights for
future work in latent reasoning.
Hybrid reasoning policy optimization (HRPO) leverages reinforcement learning to integrate latent reasoning with large language models, enhancing performance in knowledge- and reasoning-intensive tasks while maintaining interpretability.
latent reasoning autoregressive reasoning hidden states chain-of-thought (CoT) reinforcement learning (RL) hybrid reasoning HRPO token embeddings generative capabilities discrete representations continuous representations stochasticity token sampling cross-lingual patterns completion lengths