Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for
  Frozen LLMs
Large Language Models (LLMs) have demonstrated remarkable general
capabilities, but enhancing skills such as reasoning often demands substantial
computational resources and may compromise their generalization. While
Parameter-Efficient Fine-Tuning (PEFT) methods offer a more resource-conscious
alternative, they typically requires retraining for each LLM backbone due to
architectural dependencies. To address these challenges, here we propose
Universal Reasoner (UniR) - a single, lightweight, composable, and
plug-and-play reasoning module that can be used with any frozen LLM to endow it
with specialized reasoning capabilities. Specifically, UniR decomposes the
reward into a standalone reasoning module that is trained independently using
predefined rewards, effectively translating trajectory-level signals into
token-level guidance. Once trained, UniR can be combined with any frozen LLM at
inference time by simply adding its output logits to those of the LLM backbone.
This additive structure naturally enables modular composition: multiple UniR
modules trained for different tasks can be jointly applied by summing their
logits, enabling complex reasoning via composition. Experimental results on
mathematical reasoning and machine translation tasks show that UniR
significantly outperforms existing baseline fine-tuning methods using the
Llama3.2 model. Furthermore, UniR demonstrates strong weak-to-strong
generalization: reasoning modules trained on smaller models effectively guide
much larger LLMs. This makes UniR a cost-efficient, adaptable, and robust
solution for enhancing reasoning in LLMs without compromising their core
capabilities. Code is open-sourced at https://github.com/hangeol/UniR
6837fc484d14d7c8800e8b9c
Jaemin Kim
claimed_verified
2025-05-29T07:41:35.589Z
64c3732de6c3860fba66ceb0
/avatars/785783ca08687923053eac641326281f.svg
JaeminKim
user
kjm981995
6837fc484d14d7c8800e8b9d
Hangeol Chang
6837fc484d14d7c8800e8b9e
Hyunmin Hwang
6837fc484d14d7c8800e8b9f
Choonghan Kim
6837fc484d14d7c8800e8ba0
Jong Chul Ye
Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for
  Frozen LLMs
Large Language Models (LLMs) have demonstrated remarkable general
capabilities, but enhancing skills such as reasoning often demands substantial
computational resources and may compromise their generalization. While
Parameter-Efficient Fine-Tuning (PEFT) methods offer a more resource-conscious
alternative, they typically requires retraining for each LLM backbone due to
architectural dependencies. To address these challenges, here we propose
Universal Reasoner (UniR) - a single, lightweight, composable, and
plug-and-play reasoning module that can be used with any frozen LLM to endow it
with specialized reasoning capabilities. Specifically, UniR decomposes the
reward into a standalone reasoning module that is trained independently using
predefined rewards, effectively translating trajectory-level signals into
token-level guidance. Once trained, UniR can be combined with any frozen LLM at
inference time by simply adding its output logits to those of the LLM backbone.
This additive structure naturally enables modular composition: multiple UniR
modules trained for different tasks can be jointly applied by summing their
logits, enabling complex reasoning via composition. Experimental results on
mathematical reasoning and machine translation tasks show that UniR
significantly outperforms existing baseline fine-tuning methods using the
Llama3.2 model. Furthermore, UniR demonstrates strong weak-to-strong
generalization: reasoning modules trained on smaller models effectively guide
much larger LLMs. This makes UniR a cost-efficient, adaptable, and robust
solution for enhancing reasoning in LLMs without compromising their core
capabilities. Code is open-sourced at https://github.com/hangeol/UniR
UniR, a lightweight reasoning module, enhances Large Language Models with specialized reasoning abilities through modular composition, improving performance and generalization at lower computational costs.
Large Language Models Parameter-Efficient Fine-Tuning Universal Reasoner trajectory-level signals token-level guidance additive structure modular composition Llama3.2 mathematical reasoning machine translation cost-efficient adaptable robust