title: Learning from Failures in Multi-Attempt Reinforcement Learning
summary: Recent advancements in reinforcement learning (RL) for large language models
  (LLMs), exemplified by DeepSeek R1, have shown that even a simple
  question-answering task can substantially improve an LLM's reasoning
  capabilities. In this work, we extend this approach by modifying the task into
  a multi-attempt setting. Instead of generating a single response per question,
  the model is given multiple attempts, with feedback provided after incorrect
  responses. The multi-attempt task encourages the model to refine its previous
  attempts and improve search efficiency. Experimental results show that even a
  small LLM trained on a multi-attempt task achieves significantly higher
  accuracy when evaluated with more attempts, improving from 45.6% with 1 attempt
  to 52.5% with 2 attempts on the math benchmark. In contrast, the same LLM
  trained on a standard single-turn task exhibits only a marginal improvement,
  increasing from 42.3% to 43.2% when given more attempts during evaluation. The
  results indicate that, compared to the standard single-turn task, an LLM
  trained on a multi-attempt task achieves slightly better performance on math
  benchmarks while also learning to refine its responses more effectively based
  on user feedback. Full code is available at
  https://github.com/DualityRL/multi-attempt
paper_authors._id: 67ce5c7065b141ae6b0d3957
paper_authors.name: Stephen Chung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T17:39:13.472Z
paper_authors._id: 67cef8b7d9f3ce4930069e10
paper_authors.avatarUrl: /avatars/0b660e5ff13cf6cc27804bc722a70efc.svg
paper_authors.fullname: Sephen Chung
paper_authors.type: user
paper_authors.user: stephenchungmh
paper_authors._id: 67ce5c7065b141ae6b0d3958
paper_authors.name: Wenyu Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T10:27:00.543Z
paper_authors._id: 624c3d2ca19f20b197761ba9
paper_authors.avatarUrl: /avatars/7a64b81c29f4f6700fa18effc5616865.svg
paper_authors.fullname: Wenyu Du
paper_authors.type: user
paper_authors.user: wydu
paper_authors._id: 67ce5c7065b141ae6b0d3959
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T12:46:28.886Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_title: Learning from Failures in Multi-Attempt Reinforcement Learning
paper_summary: Recent advancements in reinforcement learning (RL) for large language models
  (LLMs), exemplified by DeepSeek R1, have shown that even a simple
  question-answering task can substantially improve an LLM's reasoning
  capabilities. In this work, we extend this approach by modifying the task into
  a multi-attempt setting. Instead of generating a single response per question,
  the model is given multiple attempts, with feedback provided after incorrect
  responses. The multi-attempt task encourages the model to refine its previous
  attempts and improve search efficiency. Experimental results show that even a
  small LLM trained on a multi-attempt task achieves significantly higher
  accuracy when evaluated with more attempts, improving from 45.6% with 1 attempt
  to 52.5% with 2 attempts on the math benchmark. In contrast, the same LLM
  trained on a standard single-turn task exhibits only a marginal improvement,
  increasing from 42.3% to 43.2% when given more attempts during evaluation. The
  results indicate that, compared to the standard single-turn task, an LLM
  trained on a multi-attempt task achieves slightly better performance on math
  benchmarks while also learning to refine its responses more effectively based
  on user feedback. Full code is available at
  https://github.com/DualityRL/multi-attempt
paper_ai_summary: Training large language models on a multi-attempt task with feedback improves their accuracy and response refinement compared to standard single-turn tasks.
paper_ai_keywords: reinforcement learning large language models multi-attempt setting feedback search efficiency math benchmark