title: ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent
summary: Answering complex natural language questions often necessitates multi-step
  reasoning and integrating external information. Several systems have combined
  knowledge retrieval with a large language model (LLM) to answer such questions.
  These systems, however, suffer from various failure cases, and we cannot
  directly train them end-to-end to fix such failures, as interaction with
  external knowledge is non-differentiable. To address these deficiencies, we
  define a ReAct-style LLM agent with the ability to reason and act upon external
  knowledge. We further refine the agent through a ReST-like method that
  iteratively trains on previous trajectories, employing growing-batch
  reinforcement learning with AI feedback for continuous self-improvement and
  self-distillation. Starting from a prompted large model and after just two
  iterations of the algorithm, we can produce a fine-tuned small model that
  achieves comparable performance on challenging compositional question-answering
  benchmarks with two orders of magnitude fewer parameters.
paper_authors._id: 657fb4eac7cb69069ab7be37
paper_authors.name: Renat Aksitov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T09:35:35.365Z
paper_authors._id: 6003c87f532970af8c4d3a4a
paper_authors.avatarUrl: /avatars/a4c5fbe427791d02e3cee208f22f18c4.svg
paper_authors.fullname: Renat Aksitov
paper_authors.type: user
paper_authors.user: mendor
paper_authors._id: 657fb4eac7cb69069ab7be38
paper_authors.name: Sobhan Miryoosefi
paper_authors._id: 657fb4eac7cb69069ab7be39
paper_authors.name: Zonglin Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T09:35:46.738Z
paper_authors._id: 64a7be520c36529d750714de
paper_authors.avatarUrl: /avatars/0d500be5cb7482fae2713828c8675e59.svg
paper_authors.fullname: lizonglin
paper_authors.type: user
paper_authors.user: zonglinli
paper_authors._id: 657fb4eac7cb69069ab7be3a
paper_authors.name: Daliang Li
paper_authors._id: 657fb4eac7cb69069ab7be3b
paper_authors.name: Sheila Babayan
paper_authors._id: 657fb4eac7cb69069ab7be3c
paper_authors.name: Kavya Kopparapu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T09:36:04.788Z
paper_authors._id: 655392b12bcac7af499309fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_exjMM015vU3nUdL0wb0w.png
paper_authors.fullname: Kavya Kopparapu
paper_authors.type: user
paper_authors.user: kavyakvk
paper_authors._id: 657fb4eac7cb69069ab7be3d
paper_authors.name: Zachary Fisher
paper_authors._id: 657fb4eac7cb69069ab7be3e
paper_authors.name: Ruiqi Guo
paper_authors._id: 657fb4eac7cb69069ab7be3f
paper_authors.name: Sushant Prakash
paper_authors._id: 657fb4eac7cb69069ab7be40
paper_authors.name: Pranesh Srinivasan
paper_authors._id: 657fb4eac7cb69069ab7be41
paper_authors.name: Manzil Zaheer
paper_authors._id: 657fb4eac7cb69069ab7be42
paper_authors.name: Felix Yu
paper_authors._id: 657fb4eac7cb69069ab7be43
paper_authors.name: Sanjiv Kumar
paper_title: ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent
paper_summary: Answering complex natural language questions often necessitates multi-step
  reasoning and integrating external information. Several systems have combined
  knowledge retrieval with a large language model (LLM) to answer such questions.
  These systems, however, suffer from various failure cases, and we cannot
  directly train them end-to-end to fix such failures, as interaction with
  external knowledge is non-differentiable. To address these deficiencies, we
  define a ReAct-style LLM agent with the ability to reason and act upon external
  knowledge. We further refine the agent through a ReST-like method that
  iteratively trains on previous trajectories, employing growing-batch
  reinforcement learning with AI feedback for continuous self-improvement and
  self-distillation. Starting from a prompted large model and after just two
  iterations of the algorithm, we can produce a fine-tuned small model that
  achieves comparable performance on challenging compositional question-answering
  benchmarks with two orders of magnitude fewer parameters.
paper_ai_summary: A ReAct-style LLM agent with iterative training using reinforcement learning and self-distillation improves complex question-answering performance with fewer parameters.
paper_ai_keywords: ReAct-style large language model knowledge retrieval non-differentiable ReST-like growing-batch reinforcement learning AI feedback continuous self-improvement self-distillation compositional question-answering fine-tuned parameter-efficient