Improving Language Model Negotiation with Self-Play and In-Context
  Learning from AI Feedback
We study whether multiple large language models (LLMs) can autonomously
improve each other in a negotiation game by playing, reflecting, and
criticizing. We are interested in this question because if LLMs were able to
improve each other, it would imply the possibility of creating strong AI agents
with minimal human intervention. We ask two LLMs to negotiate with each other,
playing the roles of a buyer and a seller, respectively. They aim to reach a
deal with the buyer targeting a lower price and the seller a higher one. A
third language model, playing the critic, provides feedback to a player to
improve the player's negotiation strategies. We let the two agents play
multiple rounds, using previous negotiation history and AI feedback as
in-context demonstrations to improve the model's negotiation strategy
iteratively. We use different LLMs (GPT and Claude) for different roles and use
the deal price as the evaluation metric. Our experiments reveal multiple
intriguing findings: (1) Only a subset of the language models we consider can
self-play and improve the deal price from AI feedback, weaker models either do
not understand the game's rules or cannot incorporate AI feedback for further
improvement. (2) Models' abilities to learn from the feedback differ when
playing different roles. For example, it is harder for Claude-instant to
improve as the buyer than as the seller. (3) When unrolling the game to
multiple rounds, stronger agents can consistently improve their performance by
meaningfully using previous experiences and iterative AI feedback, yet have a
higher risk of breaking the deal. We hope our work provides insightful initial
explorations of having models autonomously improve each other with game playing
and AI feedback.
64658738e22ed5ceaa6e3178
Yao Fu
claimed_verified
2023-05-30T11:11:26.301Z
636c929d5ef5a1a07db4e785
/avatars/1bb48dbba964f7fa84036586032684cf.svg
Yao Fu
user
yaofu
64658738e22ed5ceaa6e3179
Hao Peng
extracted_pending
2023-05-18T02:02:34.149Z
644c129af30d4ab8ea0f5924
/avatars/4c3322c5fc1f361b5aa59acc3e475f15.svg
Hao Peng
user
hapeng
64658738e22ed5ceaa6e317a
Tushar Khot
extracted_confirmed
2023-05-18T17:59:19.380Z
62cdf2dbaac2c91c95581830
https://cdn-avatars.huggingface.co/v1/production/uploads/62cdf2dbaac2c91c95581830/N3NuO7j53qkauJOf3kkRS.jpeg
Tushar Khot
user
tusharkhot
64658738e22ed5ceaa6e317b
Mirella Lapata
Improving Language Model Negotiation with Self-Play and In-Context
  Learning from AI Feedback
We study whether multiple large language models (LLMs) can autonomously
improve each other in a negotiation game by playing, reflecting, and
criticizing. We are interested in this question because if LLMs were able to
improve each other, it would imply the possibility of creating strong AI agents
with minimal human intervention. We ask two LLMs to negotiate with each other,
playing the roles of a buyer and a seller, respectively. They aim to reach a
deal with the buyer targeting a lower price and the seller a higher one. A
third language model, playing the critic, provides feedback to a player to
improve the player's negotiation strategies. We let the two agents play
multiple rounds, using previous negotiation history and AI feedback as
in-context demonstrations to improve the model's negotiation strategy
iteratively. We use different LLMs (GPT and Claude) for different roles and use
the deal price as the evaluation metric. Our experiments reveal multiple
intriguing findings: (1) Only a subset of the language models we consider can
self-play and improve the deal price from AI feedback, weaker models either do
not understand the game's rules or cannot incorporate AI feedback for further
improvement. (2) Models' abilities to learn from the feedback differ when
playing different roles. For example, it is harder for Claude-instant to
improve as the buyer than as the seller. (3) When unrolling the game to
multiple rounds, stronger agents can consistently improve their performance by
meaningfully using previous experiences and iterative AI feedback, yet have a
higher risk of breaking the deal. We hope our work provides insightful initial
explorations of having models autonomously improve each other with game playing
and AI feedback.
Two language models improve their negotiation strategies through iterative gameplay and AI feedback, revealing insights into self-improvement and role-specific learning.
large language models negotiation game buyer seller critic AI feedback self-play iterative improvement evaluation metric game playing