A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning
Multi-turn problem solving is critical yet challenging for Large Reasoning
Models (LRMs) to reflect on their reasoning and revise from feedback. Existing
Reinforcement Learning (RL) methods train large reasoning models on a
single-turn paradigm with verifiable rewards. However, we observe that models
trained with existing RL paradigms often lose their ability to solve problems
across multiple turns and struggle to revise answers based on contextual
feedback, leading to repetitive responses. We ask: can LRMs learn to reflect
their answers in a multi-turn context? In this work, we find that training
models with multi-turn RL using only unary feedback (e.g., "Let's try again")
after wrong answers can improve both single-turn performance and multi-turn
reasoning. We introduce Unary Feedback as Observation (UFO) for reinforcement
learning, which uses minimal yet common unary user feedback during iterative
problem solving. It can be easily applied to existing single-turn RL training
setups. Experimental results show that RL training with UFO keeps single-turn
performance and improves multi-turn reasoning accuracy by up to 14%, enabling
language models to better react to feedback in multi-turn problem solving. To
further minimize the number of turns needed for a correct answer while
encouraging diverse reasoning when mistakes occur, we design reward structures
that guide models to produce careful and deliberate answers in each turn. Code:
https://github.com/lichengliu03/unary-feedback
687f5b5033947f780d9b4bf8
Licheng Liu
claimed_verified
2025-07-22T13:05:24.744Z
661caf2f6158c41091b3954e
https://cdn-avatars.huggingface.co/v1/production/uploads/661caf2f6158c41091b3954e/zrAqbiL4u1l6_Loea00_k.jpeg
Licheng Liu
user
LichengLiu03
687f5b5033947f780d9b4bf9
Zihan Wang
687f5b5033947f780d9b4bfa
Linjie Li
687f5b5033947f780d9b4bfb
Chenwei Xu
687f5b5033947f780d9b4bfc
Yiping Lu
687f5b5033947f780d9b4bfd
Han Liu
687f5b5033947f780d9b4bfe
Avirup Sil
687f5b5033947f780d9b4bff
Manling Li
A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning
Multi-turn problem solving is critical yet challenging for Large Reasoning
Models (LRMs) to reflect on their reasoning and revise from feedback. Existing
Reinforcement Learning (RL) methods train large reasoning models on a
single-turn paradigm with verifiable rewards. However, we observe that models
trained with existing RL paradigms often lose their ability to solve problems
across multiple turns and struggle to revise answers based on contextual
feedback, leading to repetitive responses. We ask: can LRMs learn to reflect
their answers in a multi-turn context? In this work, we find that training
models with multi-turn RL using only unary feedback (e.g., "Let's try again")
after wrong answers can improve both single-turn performance and multi-turn
reasoning. We introduce Unary Feedback as Observation (UFO) for reinforcement
learning, which uses minimal yet common unary user feedback during iterative
problem solving. It can be easily applied to existing single-turn RL training
setups. Experimental results show that RL training with UFO keeps single-turn
performance and improves multi-turn reasoning accuracy by up to 14%, enabling
language models to better react to feedback in multi-turn problem solving. To
further minimize the number of turns needed for a correct answer while
encouraging diverse reasoning when mistakes occur, we design reward structures
that guide models to produce careful and deliberate answers in each turn. Code:
https://github.com/lichengliu03/unary-feedback
Training large reasoning models with multi-turn reinforcement learning using unary feedback improves both single-turn performance and multi-turn reasoning accuracy.
Reinforcement Learning RL multi-turn RL unary feedback single-turn performance multi-turn reasoning UFO reward structures