title: Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for
    Frozen LLMs
summary: Large Language Models (LLMs) have demonstrated remarkable general
  capabilities, but enhancing skills such as reasoning often demands substantial
  computational resources and may compromise their generalization. While
  Parameter-Efficient Fine-Tuning (PEFT) methods offer a more resource-conscious
  alternative, they typically requires retraining for each LLM backbone due to
  architectural dependencies. To address these challenges, here we propose
  Universal Reasoner (UniR) - a single, lightweight, composable, and
  plug-and-play reasoning module that can be used with any frozen LLM to endow it
  with specialized reasoning capabilities. Specifically, UniR decomposes the
  reward into a standalone reasoning module that is trained independently using
  predefined rewards, effectively translating trajectory-level signals into
  token-level guidance. Once trained, UniR can be combined with any frozen LLM at
  inference time by simply adding its output logits to those of the LLM backbone.
  This additive structure naturally enables modular composition: multiple UniR
  modules trained for different tasks can be jointly applied by summing their
  logits, enabling complex reasoning via composition. Experimental results on
  mathematical reasoning and machine translation tasks show that UniR
  significantly outperforms existing baseline fine-tuning methods using the
  Llama3.2 model. Furthermore, UniR demonstrates strong weak-to-strong
  generalization: reasoning modules trained on smaller models effectively guide
  much larger LLMs. This makes UniR a cost-efficient, adaptable, and robust
  solution for enhancing reasoning in LLMs without compromising their core
  capabilities. Code is open-sourced at https://github.com/hangeol/UniR
paper_authors._id: 6837fc484d14d7c8800e8b9c
paper_authors.name: Jaemin Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:41:35.589Z
paper_authors._id: 64c3732de6c3860fba66ceb0
paper_authors.avatarUrl: /avatars/785783ca08687923053eac641326281f.svg
paper_authors.fullname: JaeminKim
paper_authors.type: user
paper_authors.user: kjm981995
paper_authors._id: 6837fc484d14d7c8800e8b9d
paper_authors.name: Hangeol Chang
paper_authors._id: 6837fc484d14d7c8800e8b9e
paper_authors.name: Hyunmin Hwang
paper_authors._id: 6837fc484d14d7c8800e8b9f
paper_authors.name: Choonghan Kim
paper_authors._id: 6837fc484d14d7c8800e8ba0
paper_authors.name: Jong Chul Ye
paper_title: Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for
    Frozen LLMs
paper_summary: Large Language Models (LLMs) have demonstrated remarkable general
  capabilities, but enhancing skills such as reasoning often demands substantial
  computational resources and may compromise their generalization. While
  Parameter-Efficient Fine-Tuning (PEFT) methods offer a more resource-conscious
  alternative, they typically requires retraining for each LLM backbone due to
  architectural dependencies. To address these challenges, here we propose
  Universal Reasoner (UniR) - a single, lightweight, composable, and
  plug-and-play reasoning module that can be used with any frozen LLM to endow it
  with specialized reasoning capabilities. Specifically, UniR decomposes the
  reward into a standalone reasoning module that is trained independently using
  predefined rewards, effectively translating trajectory-level signals into
  token-level guidance. Once trained, UniR can be combined with any frozen LLM at
  inference time by simply adding its output logits to those of the LLM backbone.
  This additive structure naturally enables modular composition: multiple UniR
  modules trained for different tasks can be jointly applied by summing their
  logits, enabling complex reasoning via composition. Experimental results on
  mathematical reasoning and machine translation tasks show that UniR
  significantly outperforms existing baseline fine-tuning methods using the
  Llama3.2 model. Furthermore, UniR demonstrates strong weak-to-strong
  generalization: reasoning modules trained on smaller models effectively guide
  much larger LLMs. This makes UniR a cost-efficient, adaptable, and robust
  solution for enhancing reasoning in LLMs without compromising their core
  capabilities. Code is open-sourced at https://github.com/hangeol/UniR
paper_ai_summary: UniR, a lightweight reasoning module, enhances Large Language Models with specialized reasoning abilities through modular composition, improving performance and generalization at lower computational costs.
paper_ai_keywords: Large Language Models Parameter-Efficient Fine-Tuning Universal Reasoner trajectory-level signals token-level guidance additive structure modular composition Llama3.2 mathematical reasoning machine translation cost-efficient adaptable robust