ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent
Answering complex natural language questions often necessitates multi-step
reasoning and integrating external information. Several systems have combined
knowledge retrieval with a large language model (LLM) to answer such questions.
These systems, however, suffer from various failure cases, and we cannot
directly train them end-to-end to fix such failures, as interaction with
external knowledge is non-differentiable. To address these deficiencies, we
define a ReAct-style LLM agent with the ability to reason and act upon external
knowledge. We further refine the agent through a ReST-like method that
iteratively trains on previous trajectories, employing growing-batch
reinforcement learning with AI feedback for continuous self-improvement and
self-distillation. Starting from a prompted large model and after just two
iterations of the algorithm, we can produce a fine-tuned small model that
achieves comparable performance on challenging compositional question-answering
benchmarks with two orders of magnitude fewer parameters.
657fb4eac7cb69069ab7be37
Renat Aksitov
admin_assigned
2023-12-18T09:35:35.365Z
6003c87f532970af8c4d3a4a
/avatars/a4c5fbe427791d02e3cee208f22f18c4.svg
Renat Aksitov
user
mendor
657fb4eac7cb69069ab7be38
Sobhan Miryoosefi
657fb4eac7cb69069ab7be39
Zonglin Li
admin_assigned
2023-12-18T09:35:46.738Z
64a7be520c36529d750714de
/avatars/0d500be5cb7482fae2713828c8675e59.svg
lizonglin
user
zonglinli
657fb4eac7cb69069ab7be3a
Daliang Li
657fb4eac7cb69069ab7be3b
Sheila Babayan
657fb4eac7cb69069ab7be3c
Kavya Kopparapu
admin_assigned
2023-12-18T09:36:04.788Z
655392b12bcac7af499309fe
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_exjMM015vU3nUdL0wb0w.png
Kavya Kopparapu
user
kavyakvk
657fb4eac7cb69069ab7be3d
Zachary Fisher
657fb4eac7cb69069ab7be3e
Ruiqi Guo
657fb4eac7cb69069ab7be3f
Sushant Prakash
657fb4eac7cb69069ab7be40
Pranesh Srinivasan
657fb4eac7cb69069ab7be41
Manzil Zaheer
657fb4eac7cb69069ab7be42
Felix Yu
657fb4eac7cb69069ab7be43
Sanjiv Kumar
ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent
Answering complex natural language questions often necessitates multi-step
reasoning and integrating external information. Several systems have combined
knowledge retrieval with a large language model (LLM) to answer such questions.
These systems, however, suffer from various failure cases, and we cannot
directly train them end-to-end to fix such failures, as interaction with
external knowledge is non-differentiable. To address these deficiencies, we
define a ReAct-style LLM agent with the ability to reason and act upon external
knowledge. We further refine the agent through a ReST-like method that
iteratively trains on previous trajectories, employing growing-batch
reinforcement learning with AI feedback for continuous self-improvement and
self-distillation. Starting from a prompted large model and after just two
iterations of the algorithm, we can produce a fine-tuned small model that
achieves comparable performance on challenging compositional question-answering
benchmarks with two orders of magnitude fewer parameters.
A ReAct-style LLM agent with iterative training using reinforcement learning and self-distillation improves complex question-answering performance with fewer parameters.
ReAct-style large language model knowledge retrieval non-differentiable ReST-like growing-batch reinforcement learning AI feedback continuous self-improvement self-distillation compositional question-answering fine-tuned parameter-efficient