title: Language Models are Hidden Reasoners: Unlocking Latent Reasoning
    Capabilities via Self-Rewarding
summary: Large language models (LLMs) have shown impressive capabilities, but still
  struggle with complex reasoning tasks requiring multiple steps. While
  prompt-based methods like Chain-of-Thought (CoT) can improve LLM reasoning at
  inference time, optimizing reasoning capabilities during training remains
  challenging. We introduce LaTent Reasoning Optimization (LaTRO), a principled
  framework that formulates reasoning as sampling from a latent distribution and
  optimizes it via variational approaches. LaTRO enables LLMs to concurrently
  improve both their reasoning process and ability to evaluate reasoning quality,
  without requiring external feedback or reward models. We validate LaTRO through
  experiments on GSM8K and ARC-Challenge datasets using multiple model
  architectures. On GSM8K, LaTRO improves zero-shot accuracy by an average of
  12.5% over base models and 9.6% over supervised fine-tuning across
  Phi-3.5-mini, Mistral-7B, and Llama-3.1-8B. Our findings suggest that
  pre-trained LLMs possess latent reasoning capabilities that can be unlocked and
  enhanced through our proposed optimization approach in a self-improvement
  manner. The code of LaTRO is available at
  https://github.com/SalesforceAIResearch/LaTRO.
paper_authors._id: 6732834b2c3d3a050a86288a
paper_authors.name: Haolin Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-11T22:21:00.560Z
paper_authors._id: 65e0c21ebd0c59bbf271004c
paper_authors.avatarUrl: /avatars/3210f41fd433d798fd7857c943288625.svg
paper_authors.fullname: Haolin Chen
paper_authors.type: user
paper_authors.user: hlnchen
paper_authors._id: 6732834b2c3d3a050a86288b
paper_authors.name: Yihao Feng
paper_authors._id: 6732834b2c3d3a050a86288c
paper_authors.name: Zuxin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:27:25.665Z
paper_authors._id: 638e971f2cc490759fe9210e
paper_authors.avatarUrl: /avatars/9578d0badc084d340c028a59e571abd3.svg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: Zuxin
paper_authors._id: 6732834b2c3d3a050a86288d
paper_authors.name: Weiran Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:27:23.471Z
paper_authors._id: 632cdea254e2c512c8f95b12
paper_authors.avatarUrl: /avatars/a6d06cdd75861ae7d589f1343d81a5c5.svg
paper_authors.fullname: Weiran Yao
paper_authors.type: user
paper_authors.user: weirayao
paper_authors._id: 6732834b2c3d3a050a86288e
paper_authors.name: Akshara Prabhakar
paper_authors._id: 6732834b2c3d3a050a86288f
paper_authors.name: Shelby Heinecke
paper_authors._id: 6732834b2c3d3a050a862890
paper_authors.name: Ricky Ho
paper_authors._id: 6732834b2c3d3a050a862891
paper_authors.name: Phil Mui
paper_authors._id: 6732834b2c3d3a050a862892
paper_authors.name: Silvio Savarese
paper_authors._id: 6732834b2c3d3a050a862893
paper_authors.name: Caiming Xiong
paper_authors._id: 6732834b2c3d3a050a862894
paper_authors.name: Huan Wang
paper_title: Language Models are Hidden Reasoners: Unlocking Latent Reasoning
    Capabilities via Self-Rewarding
paper_summary: Large language models (LLMs) have shown impressive capabilities, but still
  struggle with complex reasoning tasks requiring multiple steps. While
  prompt-based methods like Chain-of-Thought (CoT) can improve LLM reasoning at
  inference time, optimizing reasoning capabilities during training remains
  challenging. We introduce LaTent Reasoning Optimization (LaTRO), a principled
  framework that formulates reasoning as sampling from a latent distribution and
  optimizes it via variational approaches. LaTRO enables LLMs to concurrently
  improve both their reasoning process and ability to evaluate reasoning quality,
  without requiring external feedback or reward models. We validate LaTRO through
  experiments on GSM8K and ARC-Challenge datasets using multiple model
  architectures. On GSM8K, LaTRO improves zero-shot accuracy by an average of
  12.5% over base models and 9.6% over supervised fine-tuning across
  Phi-3.5-mini, Mistral-7B, and Llama-3.1-8B. Our findings suggest that
  pre-trained LLMs possess latent reasoning capabilities that can be unlocked and
  enhanced through our proposed optimization approach in a self-improvement
  manner. The code of LaTRO is available at
  https://github.com/SalesforceAIResearch/LaTRO.
paper_ai_keywords: Chain-of-Thought (CoT) LaTent Reasoning Optimization (LaTRO) latent distribution variational approaches zero-shot accuracy GSM8K ARC-Challenge Phi-3.5-mini Mistral-7B Llama-3.1-8B pre-trained LLMs latent reasoning capabilities self-improvement manner