Beyond the Trade-off: Self-Supervised Reinforcement Learning for
  Reasoning Models' Instruction Following
Reasoning models excel in complex problem solving but exhibit a concerning
trade off between reasoning capabilities and instruction following abilities.
Existing approaches for improving instruction following rely on stronger
external models, creating methodological bottlenecks and practical limitations
including increased costs and accessibility constraints. We propose a
self-supervised RL framework that leverages reasoning models' own internal
signals to improve instruction following capabilities without external
supervision. Extensive experiments demonstrate that our framework significantly
improves instruction following capabilities while maintaining reasoning
performance, offering a scalable and cost-effective approach to enhance
instruction following in reasoning models. The data and code are publicly
available at https://github.com/Rainier-rq/verl-if.
6891804cf01a094725f8350e
Qingyu Ren
6891804cf01a094725f8350f
Qianyu He
6891804cf01a094725f83510
Bowei Zhang
6891804cf01a094725f83511
Jie Zeng
6891804cf01a094725f83512
Jiaqing Liang
6891804cf01a094725f83513
Yanghua Xiao
6891804cf01a094725f83514
Weikang Zhou
6891804cf01a094725f83515
Zeye Sun
6891804cf01a094725f83516
Fei Yu
Beyond the Trade-off: Self-Supervised Reinforcement Learning for
  Reasoning Models' Instruction Following
Reasoning models excel in complex problem solving but exhibit a concerning
trade off between reasoning capabilities and instruction following abilities.
Existing approaches for improving instruction following rely on stronger
external models, creating methodological bottlenecks and practical limitations
including increased costs and accessibility constraints. We propose a
self-supervised RL framework that leverages reasoning models' own internal
signals to improve instruction following capabilities without external
supervision. Extensive experiments demonstrate that our framework significantly
improves instruction following capabilities while maintaining reasoning
performance, offering a scalable and cost-effective approach to enhance
instruction following in reasoning models. The data and code are publicly
available at https://github.com/Rainier-rq/verl-if.
A self-supervised RL framework enhances instruction following in reasoning models without external supervision, maintaining reasoning performance and offering scalability and cost-effectiveness.
self-supervised RL reasoning models instruction following internal signals scalability cost-effectiveness