title: Retroformer: Retrospective Large Language Agents with Policy Gradient
    Optimization
summary: Recent months have seen the emergence of a powerful new trend in which large
  language models (LLMs) are augmented to become autonomous language agents
  capable of performing objective oriented multi-step tasks on their own, rather
  than merely responding to queries from human users. Most existing language
  agents, however, are not optimized using environment-specific rewards. Although
  some agents enable iterative refinement through verbal feedback, they do not
  reason and plan in ways that are compatible with gradient-based learning from
  rewards. This paper introduces a principled framework for reinforcing large
  language agents by learning a retrospective model, which automatically tunes
  the language agent prompts from environment feedback through policy gradient.
  Specifically, our proposed agent architecture learns from rewards across
  multiple environments and tasks, for fine-tuning a pre-trained language model
  which refines the language agent prompt by summarizing the root cause of prior
  failed attempts and proposing action plans. Experimental results on various
  tasks demonstrate that the language agents improve over time and that our
  approach considerably outperforms baselines that do not properly leverage
  gradients from the environment. This demonstrates that using policy gradient
  optimization to improve language agents, for which we believe our work is one
  of the first, seems promising and can be applied to optimize other models in
  the agent architecture to enhance agent performances over time.
paper_authors._id: 64d0642d4dfd5df70735dfd0
paper_authors.name: Weiran Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:41:22.890Z
paper_authors._id: 632cdea254e2c512c8f95b12
paper_authors.avatarUrl: /avatars/a6d06cdd75861ae7d589f1343d81a5c5.svg
paper_authors.fullname: Weiran Yao
paper_authors.type: user
paper_authors.user: weirayao
paper_authors._id: 64d0642d4dfd5df70735dfd1
paper_authors.name: Shelby Heinecke
paper_authors._id: 64d0642d4dfd5df70735dfd2
paper_authors.name: Juan Carlos Niebles
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-01T08:45:08.339Z
paper_authors._id: 65bb26b5d61b51a50805aba2
paper_authors.avatarUrl: /avatars/966c8469f55776948a7cf42d98accff3.svg
paper_authors.fullname: Juan Carlos Niebles
paper_authors.type: user
paper_authors.user: niebles
paper_authors._id: 64d0642d4dfd5df70735dfd3
paper_authors.name: Zhiwei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-27T08:12:44.110Z
paper_authors._id: 64470130bd9e0e0edf5227d7
paper_authors.avatarUrl: /avatars/89f118f880cce3d01658b123bbbf4402.svg
paper_authors.fullname: Zhiwei Liu
paper_authors.type: user
paper_authors.user: jimzhiwei
paper_authors._id: 64d0642d4dfd5df70735dfd4
paper_authors.name: Yihao Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:42:15.636Z
paper_authors._id: 6354d31106d707b3324a3a5a
paper_authors.avatarUrl: /avatars/a662d176dfc4b809cd75aa726f24b74c.svg
paper_authors.fullname: Yihao Feng
paper_authors.type: user
paper_authors.user: yihaocs
paper_authors._id: 64d0642d4dfd5df70735dfd5
paper_authors.name: Le Xue
paper_authors._id: 64d0642d4dfd5df70735dfd6
paper_authors.name: Rithesh Murthy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T08:03:39.496Z
paper_authors._id: 60381555c58b41eeda0d6ccf
paper_authors.avatarUrl: /avatars/2c2c4b75dd0199478dc0fbc72f9bcadc.svg
paper_authors.fullname: rmurthy
paper_authors.type: user
paper_authors.user: rmurthy
paper_authors._id: 64d0642d4dfd5df70735dfd7
paper_authors.name: Zeyuan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:42:54.224Z
paper_authors._id: 632405844624707127217677
paper_authors.avatarUrl: /avatars/af5deba9ec51814bc5ccfd04f68d76d3.svg
paper_authors.fullname: Zeyuan Chen
paper_authors.type: user
paper_authors.user: bradypus
paper_authors._id: 64d0642d4dfd5df70735dfd8
paper_authors.name: Jianguo Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:43:17.675Z
paper_authors._id: 630fcbd758d83e8f64d82777
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630fcbd758d83e8f64d82777/0jwTa1MzeXZbDtJ-uXGo6.jpeg
paper_authors.fullname: Jianguo Zhang
paper_authors.type: user
paper_authors.user: jianguozhang
paper_authors._id: 64d0642d4dfd5df70735dfd9
paper_authors.name: Devansh Arpit
paper_authors._id: 64d0642d4dfd5df70735dfda
paper_authors.name: Ran Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:15:15.092Z
paper_authors._id: 6465c4c863e7e09dd02e3e1b
paper_authors.avatarUrl: /avatars/200b029184d2616f98296a2c212f0785.svg
paper_authors.fullname: Ran Xu
paper_authors.type: user
paper_authors.user: xurantju
paper_authors._id: 64d0642d4dfd5df70735dfdb
paper_authors.name: Phil Mui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:44:04.179Z
paper_authors._id: 63169c6618490c4b7a550ee2
paper_authors.avatarUrl: /avatars/c71726db2fdfea9a1c107eb16928517d.svg
paper_authors.fullname: Phil Mui
paper_authors.type: user
paper_authors.user: philmui
paper_authors._id: 64d0642d4dfd5df70735dfdc
paper_authors.name: Huan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-08T01:20:30.599Z
paper_authors._id: 64b96d0bea67d88ebec0f983
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b96d0bea67d88ebec0f983/SnZwF6_ABRXhU64CoFoZm.jpeg
paper_authors.fullname: Huan Wang
paper_authors.type: user
paper_authors.user: joyousprince
paper_authors._id: 64d0642d4dfd5df70735dfdd
paper_authors.name: Caiming Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:45:13.585Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_authors._id: 64d0642d4dfd5df70735dfde
paper_authors.name: Silvio Savarese
paper_title: Retroformer: Retrospective Large Language Agents with Policy Gradient
    Optimization
paper_summary: Recent months have seen the emergence of a powerful new trend in which large
  language models (LLMs) are augmented to become autonomous language agents
  capable of performing objective oriented multi-step tasks on their own, rather
  than merely responding to queries from human users. Most existing language
  agents, however, are not optimized using environment-specific rewards. Although
  some agents enable iterative refinement through verbal feedback, they do not
  reason and plan in ways that are compatible with gradient-based learning from
  rewards. This paper introduces a principled framework for reinforcing large
  language agents by learning a retrospective model, which automatically tunes
  the language agent prompts from environment feedback through policy gradient.
  Specifically, our proposed agent architecture learns from rewards across
  multiple environments and tasks, for fine-tuning a pre-trained language model
  which refines the language agent prompt by summarizing the root cause of prior
  failed attempts and proposing action plans. Experimental results on various
  tasks demonstrate that the language agents improve over time and that our
  approach considerably outperforms baselines that do not properly leverage
  gradients from the environment. This demonstrates that using policy gradient
  optimization to improve language agents, for which we believe our work is one
  of the first, seems promising and can be applied to optimize other models in
  the agent architecture to enhance agent performances over time.
paper_ai_keywords: large language models (LLMs) autonomous language agents multi-step tasks environment-specific rewards iterative refinement verbal feedback policy gradient retrospective model language agent prompts root cause action plans MTEB fine-tuning gradient-based learning policy gradient optimization