Leveraging Implicit Feedback from Deployment Data in Dialogue
We study improving social conversational agents by learning from natural
dialogue between users and a deployed model, without extra annotations. To
implicitly measure the quality of a machine-generated utterance, we leverage
signals like user response length, sentiment and reaction of the future human
utterances in the collected dialogue episodes. Our experiments use the publicly
released deployment data from BlenderBot (Xu et al., 2023). Human evaluation
indicates improvements in our new models over baseline responses; however, we
find that some proxy signals can lead to more generations with undesirable
properties as well. For example, optimizing for conversation length can lead to
more controversial or unfriendly generations compared to the baseline, whereas
optimizing for positive sentiment or reaction can decrease these behaviors.
64c24d80f6a0a34ddc5da9df
Richard Yuanzhe Pang
claimed_verified
2023-08-11T02:11:45.109Z
64d50fa59a6a7ae984d90852
/avatars/26b78c151e707e3f9a9516c877d18ff7.svg
Richard Pang
user
yzpang
64c24d80f6a0a34ddc5da9e0
Stephen Roller
admin_assigned
2023-07-27T11:28:36.660Z
627d2ab9cecd686d4cd8f922
/avatars/3a063adebd97b0a7762edca3a4a92218.svg
Stephen Roller
user
stephenroller
64c24d80f6a0a34ddc5da9e1
Kyunghyun Cho
admin_assigned
2023-07-27T11:27:10.196Z
649c1b8485b05ee02723334a
/avatars/29080f550fdb471c4d2daace86aff3b0.svg
Kyunghyun Cho
user
kyunghyuncho
64c24d80f6a0a34ddc5da9e2
He He
64c24d80f6a0a34ddc5da9e3
Jason Weston
admin_assigned
2023-07-27T11:27:52.357Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
Leveraging Implicit Feedback from Deployment Data in Dialogue
We study improving social conversational agents by learning from natural
dialogue between users and a deployed model, without extra annotations. To
implicitly measure the quality of a machine-generated utterance, we leverage
signals like user response length, sentiment and reaction of the future human
utterances in the collected dialogue episodes. Our experiments use the publicly
released deployment data from BlenderBot (Xu et al., 2023). Human evaluation
indicates improvements in our new models over baseline responses; however, we
find that some proxy signals can lead to more generations with undesirable
properties as well. For example, optimizing for conversation length can lead to
more controversial or unfriendly generations compared to the baseline, whereas
optimizing for positive sentiment or reaction can decrease these behaviors.
Improvements in social conversational agents are achieved by learning from user interactions without annotations, using signals like response length, sentiment, and reactions to measure utterance quality.
conversational agents dialogue BlenderBot user response length sentiment reactions