Incentivizing Reasoning for Advanced Instruction-Following of Large
  Language Models
Existing large language models (LLMs) face challenges of following complex
instructions, especially when multiple constraints are present and organized in
paralleling, chaining, and branching structures. One intuitive solution, namely
chain-of-thought (CoT), is expected to universally improve capabilities of
LLMs. However, we find that the vanilla CoT exerts a negative impact on
performance due to its superficial reasoning pattern of simply paraphrasing the
instructions. It fails to peel back the compositions of constraints for
identifying their relationship across hierarchies of types and dimensions. To
this end, we propose a systematic method to boost LLMs in dealing with complex
instructions via incentivizing reasoning for test-time compute scaling. First,
we stem from the decomposition of complex instructions under existing
taxonomies and propose a reproducible data acquisition method. Second, we
exploit reinforcement learning (RL) with verifiable rule-centric reward signals
to cultivate reasoning specifically for instruction following. We address the
shallow, non-essential nature of reasoning under complex instructions via
sample-wise contrast for superior CoT enforcement. We also exploit behavior
cloning of experts to facilitate steady distribution shift from fast-thinking
LLMs to skillful reasoners. Extensive evaluations on seven comprehensive
benchmarks confirm the validity of the proposed method, where a 1.5B LLM
achieves 11.74% gains with performance comparable to a 8B LLM. Codes and data
are available at https://github.com/yuleiqin/RAIF.
683e6aa057738c5cc3616d70
Yulei Qin
extracted_confirmed
2025-06-19T08:56:37.354Z
6390525c00fb8ec4a424e0ff
/avatars/4302571e2ef4a9875491221aa630a329.svg
Yulei Qin
user
yolay
683e6aa057738c5cc3616d71
Gang Li
claimed_verified
2025-06-03T12:33:53.264Z
648c17fc17c7ceb9b4200d31
/avatars/abc5b025dd682935056c33a9e1e362f0.svg
Gang Li
user
ligang-cs
683e6aa057738c5cc3616d72
Zongyi Li
683e6aa057738c5cc3616d73
Zihan Xu
683e6aa057738c5cc3616d74
Yuchen Shi
683e6aa057738c5cc3616d75
Zhekai Lin
683e6aa057738c5cc3616d76
Xiao Cui
683e6aa057738c5cc3616d77
Ke Li
683e6aa057738c5cc3616d78
Xing Sun
Incentivizing Reasoning for Advanced Instruction-Following of Large
  Language Models
Existing large language models (LLMs) face challenges of following complex
instructions, especially when multiple constraints are present and organized in
paralleling, chaining, and branching structures. One intuitive solution, namely
chain-of-thought (CoT), is expected to universally improve capabilities of
LLMs. However, we find that the vanilla CoT exerts a negative impact on
performance due to its superficial reasoning pattern of simply paraphrasing the
instructions. It fails to peel back the compositions of constraints for
identifying their relationship across hierarchies of types and dimensions. To
this end, we propose a systematic method to boost LLMs in dealing with complex
instructions via incentivizing reasoning for test-time compute scaling. First,
we stem from the decomposition of complex instructions under existing
taxonomies and propose a reproducible data acquisition method. Second, we
exploit reinforcement learning (RL) with verifiable rule-centric reward signals
to cultivate reasoning specifically for instruction following. We address the
shallow, non-essential nature of reasoning under complex instructions via
sample-wise contrast for superior CoT enforcement. We also exploit behavior
cloning of experts to facilitate steady distribution shift from fast-thinking
LLMs to skillful reasoners. Extensive evaluations on seven comprehensive
benchmarks confirm the validity of the proposed method, where a 1.5B LLM
achieves 11.74% gains with performance comparable to a 8B LLM. Codes and data
are available at https://github.com/yuleiqin/RAIF.
A method is proposed to enhance large language models in handling complex instructions through incentivized reasoning and reinforcement learning, improving performance and reducing computational load.
chain-of-thought (CoT) reinforcement learning (RL) rule-centric reward signals sample-wise contrast behavior cloning instruction following decomposition of complex instructions