title: Improving Language Model Negotiation with Self-Play and In-Context
    Learning from AI Feedback
summary: We study whether multiple large language models (LLMs) can autonomously
  improve each other in a negotiation game by playing, reflecting, and
  criticizing. We are interested in this question because if LLMs were able to
  improve each other, it would imply the possibility of creating strong AI agents
  with minimal human intervention. We ask two LLMs to negotiate with each other,
  playing the roles of a buyer and a seller, respectively. They aim to reach a
  deal with the buyer targeting a lower price and the seller a higher one. A
  third language model, playing the critic, provides feedback to a player to
  improve the player's negotiation strategies. We let the two agents play
  multiple rounds, using previous negotiation history and AI feedback as
  in-context demonstrations to improve the model's negotiation strategy
  iteratively. We use different LLMs (GPT and Claude) for different roles and use
  the deal price as the evaluation metric. Our experiments reveal multiple
  intriguing findings: (1) Only a subset of the language models we consider can
  self-play and improve the deal price from AI feedback, weaker models either do
  not understand the game's rules or cannot incorporate AI feedback for further
  improvement. (2) Models' abilities to learn from the feedback differ when
  playing different roles. For example, it is harder for Claude-instant to
  improve as the buyer than as the seller. (3) When unrolling the game to
  multiple rounds, stronger agents can consistently improve their performance by
  meaningfully using previous experiences and iterative AI feedback, yet have a
  higher risk of breaking the deal. We hope our work provides insightful initial
  explorations of having models autonomously improve each other with game playing
  and AI feedback.
paper_authors._id: 64658738e22ed5ceaa6e3178
paper_authors.name: Yao Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T11:11:26.301Z
paper_authors._id: 636c929d5ef5a1a07db4e785
paper_authors.avatarUrl: /avatars/1bb48dbba964f7fa84036586032684cf.svg
paper_authors.fullname: Yao Fu
paper_authors.type: user
paper_authors.user: yaofu
paper_authors._id: 64658738e22ed5ceaa6e3179
paper_authors.name: Hao Peng
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-05-18T02:02:34.149Z
paper_authors._id: 644c129af30d4ab8ea0f5924
paper_authors.avatarUrl: /avatars/4c3322c5fc1f361b5aa59acc3e475f15.svg
paper_authors.fullname: Hao Peng
paper_authors.type: user
paper_authors.user: hapeng
paper_authors._id: 64658738e22ed5ceaa6e317a
paper_authors.name: Tushar Khot
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-18T17:59:19.380Z
paper_authors._id: 62cdf2dbaac2c91c95581830
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cdf2dbaac2c91c95581830/N3NuO7j53qkauJOf3kkRS.jpeg
paper_authors.fullname: Tushar Khot
paper_authors.type: user
paper_authors.user: tusharkhot
paper_authors._id: 64658738e22ed5ceaa6e317b
paper_authors.name: Mirella Lapata
paper_title: Improving Language Model Negotiation with Self-Play and In-Context
    Learning from AI Feedback
paper_summary: We study whether multiple large language models (LLMs) can autonomously
  improve each other in a negotiation game by playing, reflecting, and
  criticizing. We are interested in this question because if LLMs were able to
  improve each other, it would imply the possibility of creating strong AI agents
  with minimal human intervention. We ask two LLMs to negotiate with each other,
  playing the roles of a buyer and a seller, respectively. They aim to reach a
  deal with the buyer targeting a lower price and the seller a higher one. A
  third language model, playing the critic, provides feedback to a player to
  improve the player's negotiation strategies. We let the two agents play
  multiple rounds, using previous negotiation history and AI feedback as
  in-context demonstrations to improve the model's negotiation strategy
  iteratively. We use different LLMs (GPT and Claude) for different roles and use
  the deal price as the evaluation metric. Our experiments reveal multiple
  intriguing findings: (1) Only a subset of the language models we consider can
  self-play and improve the deal price from AI feedback, weaker models either do
  not understand the game's rules or cannot incorporate AI feedback for further
  improvement. (2) Models' abilities to learn from the feedback differ when
  playing different roles. For example, it is harder for Claude-instant to
  improve as the buyer than as the seller. (3) When unrolling the game to
  multiple rounds, stronger agents can consistently improve their performance by
  meaningfully using previous experiences and iterative AI feedback, yet have a
  higher risk of breaking the deal. We hope our work provides insightful initial
  explorations of having models autonomously improve each other with game playing
  and AI feedback.
paper_ai_summary: Two language models improve their negotiation strategies through iterative gameplay and AI feedback, revealing insights into self-improvement and role-specific learning.
paper_ai_keywords: large language models negotiation game buyer seller critic AI feedback self-play iterative improvement evaluation metric game playing