Retroformer: Retrospective Large Language Agents with Policy Gradient
  Optimization
Recent months have seen the emergence of a powerful new trend in which large
language models (LLMs) are augmented to become autonomous language agents
capable of performing objective oriented multi-step tasks on their own, rather
than merely responding to queries from human users. Most existing language
agents, however, are not optimized using environment-specific rewards. Although
some agents enable iterative refinement through verbal feedback, they do not
reason and plan in ways that are compatible with gradient-based learning from
rewards. This paper introduces a principled framework for reinforcing large
language agents by learning a retrospective model, which automatically tunes
the language agent prompts from environment feedback through policy gradient.
Specifically, our proposed agent architecture learns from rewards across
multiple environments and tasks, for fine-tuning a pre-trained language model
which refines the language agent prompt by summarizing the root cause of prior
failed attempts and proposing action plans. Experimental results on various
tasks demonstrate that the language agents improve over time and that our
approach considerably outperforms baselines that do not properly leverage
gradients from the environment. This demonstrates that using policy gradient
optimization to improve language agents, for which we believe our work is one
of the first, seems promising and can be applied to optimize other models in
the agent architecture to enhance agent performances over time.
64d0642d4dfd5df70735dfd0
Weiran Yao
admin_assigned
2023-08-07T12:41:22.890Z
632cdea254e2c512c8f95b12
/avatars/a6d06cdd75861ae7d589f1343d81a5c5.svg
Weiran Yao
user
weirayao
64d0642d4dfd5df70735dfd1
Shelby Heinecke
64d0642d4dfd5df70735dfd2
Juan Carlos Niebles
claimed_verified
2024-02-01T08:45:08.339Z
65bb26b5d61b51a50805aba2
/avatars/966c8469f55776948a7cf42d98accff3.svg
Juan Carlos Niebles
user
niebles
64d0642d4dfd5df70735dfd3
Zhiwei Liu
claimed_verified
2024-02-27T08:12:44.110Z
64470130bd9e0e0edf5227d7
/avatars/89f118f880cce3d01658b123bbbf4402.svg
Zhiwei Liu
user
jimzhiwei
64d0642d4dfd5df70735dfd4
Yihao Feng
admin_assigned
2023-08-07T12:42:15.636Z
6354d31106d707b3324a3a5a
/avatars/a662d176dfc4b809cd75aa726f24b74c.svg
Yihao Feng
user
yihaocs
64d0642d4dfd5df70735dfd5
Le Xue
64d0642d4dfd5df70735dfd6
Rithesh Murthy
claimed_verified
2024-06-28T08:03:39.496Z
60381555c58b41eeda0d6ccf
/avatars/2c2c4b75dd0199478dc0fbc72f9bcadc.svg
rmurthy
user
rmurthy
64d0642d4dfd5df70735dfd7
Zeyuan Chen
admin_assigned
2023-08-07T12:42:54.224Z
632405844624707127217677
/avatars/af5deba9ec51814bc5ccfd04f68d76d3.svg
Zeyuan Chen
user
bradypus
64d0642d4dfd5df70735dfd8
Jianguo Zhang
admin_assigned
2023-08-07T12:43:17.675Z
630fcbd758d83e8f64d82777
https://cdn-avatars.huggingface.co/v1/production/uploads/630fcbd758d83e8f64d82777/0jwTa1MzeXZbDtJ-uXGo6.jpeg
Jianguo Zhang
user
jianguozhang
64d0642d4dfd5df70735dfd9
Devansh Arpit
64d0642d4dfd5df70735dfda
Ran Xu
claimed_verified
2024-08-19T22:15:15.092Z
6465c4c863e7e09dd02e3e1b
/avatars/200b029184d2616f98296a2c212f0785.svg
Ran Xu
user
xurantju
64d0642d4dfd5df70735dfdb
Phil Mui
admin_assigned
2023-08-07T12:44:04.179Z
63169c6618490c4b7a550ee2
/avatars/c71726db2fdfea9a1c107eb16928517d.svg
Phil Mui
user
philmui
64d0642d4dfd5df70735dfdc
Huan Wang
claimed_verified
2023-08-08T01:20:30.599Z
64b96d0bea67d88ebec0f983
https://cdn-avatars.huggingface.co/v1/production/uploads/64b96d0bea67d88ebec0f983/SnZwF6_ABRXhU64CoFoZm.jpeg
Huan Wang
user
joyousprince
64d0642d4dfd5df70735dfdd
Caiming Xiong
admin_assigned
2023-08-07T12:45:13.585Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
64d0642d4dfd5df70735dfde
Silvio Savarese
Retroformer: Retrospective Large Language Agents with Policy Gradient
  Optimization
Recent months have seen the emergence of a powerful new trend in which large
language models (LLMs) are augmented to become autonomous language agents
capable of performing objective oriented multi-step tasks on their own, rather
than merely responding to queries from human users. Most existing language
agents, however, are not optimized using environment-specific rewards. Although
some agents enable iterative refinement through verbal feedback, they do not
reason and plan in ways that are compatible with gradient-based learning from
rewards. This paper introduces a principled framework for reinforcing large
language agents by learning a retrospective model, which automatically tunes
the language agent prompts from environment feedback through policy gradient.
Specifically, our proposed agent architecture learns from rewards across
multiple environments and tasks, for fine-tuning a pre-trained language model
which refines the language agent prompt by summarizing the root cause of prior
failed attempts and proposing action plans. Experimental results on various
tasks demonstrate that the language agents improve over time and that our
approach considerably outperforms baselines that do not properly leverage
gradients from the environment. This demonstrates that using policy gradient
optimization to improve language agents, for which we believe our work is one
of the first, seems promising and can be applied to optimize other models in
the agent architecture to enhance agent performances over time.
A framework using policy gradient optimization to enhance large language agents by learning from environment feedback, improving their performance over time.
large language models language agents policy gradient retrospective model environment-specific rewards gradient-based learning fine-tuning root cause analysis action plans experimental results baselines