title: Leveraging Implicit Feedback from Deployment Data in Dialogue
summary: We study improving social conversational agents by learning from natural
  dialogue between users and a deployed model, without extra annotations. To
  implicitly measure the quality of a machine-generated utterance, we leverage
  signals like user response length, sentiment and reaction of the future human
  utterances in the collected dialogue episodes. Our experiments use the publicly
  released deployment data from BlenderBot (Xu et al., 2023). Human evaluation
  indicates improvements in our new models over baseline responses; however, we
  find that some proxy signals can lead to more generations with undesirable
  properties as well. For example, optimizing for conversation length can lead to
  more controversial or unfriendly generations compared to the baseline, whereas
  optimizing for positive sentiment or reaction can decrease these behaviors.
paper_authors._id: 64c24d80f6a0a34ddc5da9df
paper_authors.name: Richard Yuanzhe Pang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-11T02:11:45.109Z
paper_authors._id: 64d50fa59a6a7ae984d90852
paper_authors.avatarUrl: /avatars/26b78c151e707e3f9a9516c877d18ff7.svg
paper_authors.fullname: Richard Pang
paper_authors.type: user
paper_authors.user: yzpang
paper_authors._id: 64c24d80f6a0a34ddc5da9e0
paper_authors.name: Stephen Roller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T11:28:36.660Z
paper_authors._id: 627d2ab9cecd686d4cd8f922
paper_authors.avatarUrl: /avatars/3a063adebd97b0a7762edca3a4a92218.svg
paper_authors.fullname: Stephen Roller
paper_authors.type: user
paper_authors.user: stephenroller
paper_authors._id: 64c24d80f6a0a34ddc5da9e1
paper_authors.name: Kyunghyun Cho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T11:27:10.196Z
paper_authors._id: 649c1b8485b05ee02723334a
paper_authors.avatarUrl: /avatars/29080f550fdb471c4d2daace86aff3b0.svg
paper_authors.fullname: Kyunghyun Cho
paper_authors.type: user
paper_authors.user: kyunghyuncho
paper_authors._id: 64c24d80f6a0a34ddc5da9e2
paper_authors.name: He He
paper_authors._id: 64c24d80f6a0a34ddc5da9e3
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T11:27:52.357Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_title: Leveraging Implicit Feedback from Deployment Data in Dialogue
paper_summary: We study improving social conversational agents by learning from natural
  dialogue between users and a deployed model, without extra annotations. To
  implicitly measure the quality of a machine-generated utterance, we leverage
  signals like user response length, sentiment and reaction of the future human
  utterances in the collected dialogue episodes. Our experiments use the publicly
  released deployment data from BlenderBot (Xu et al., 2023). Human evaluation
  indicates improvements in our new models over baseline responses; however, we
  find that some proxy signals can lead to more generations with undesirable
  properties as well. For example, optimizing for conversation length can lead to
  more controversial or unfriendly generations compared to the baseline, whereas
  optimizing for positive sentiment or reaction can decrease these behaviors.
paper_ai_summary: Improvements in social conversational agents are achieved by learning from user interactions without annotations, using signals like response length, sentiment, and reactions to measure utterance quality.
paper_ai_keywords: conversational agents dialogue BlenderBot user response length sentiment reactions