title: Improving Open Language Models by Learning from Organic Interactions
summary: We present BlenderBot 3x, an update on the conversational model BlenderBot 3,
  which is now trained using organic conversation and feedback data from
  participating users of the system in order to improve both its skills and
  safety. We are publicly releasing the participating de-identified interaction
  data for use by the research community, in order to spur further progress.
  Training models with organic data is challenging because interactions with
  people "in the wild" include both high quality conversations and feedback, as
  well as adversarial and toxic behavior. We study techniques that enable
  learning from helpful teachers while avoiding learning from people who are
  trying to trick the model into unhelpful or toxic responses. BlenderBot 3x is
  both preferred in conversation to BlenderBot 3, and is shown to produce safer
  responses in challenging situations. While our current models are still far
  from perfect, we believe further improvement can be achieved by continued use
  of the techniques explored in this work.
paper_authors._id: 64828cc35f581f451849684c
paper_authors.name: Jing Xu
paper_authors._id: 64828cc35f581f451849684d
paper_authors.name: Da Ju
paper_authors._id: 64828cc35f581f451849684e
paper_authors.name: Joshua Lane
paper_authors._id: 64828cc35f581f451849684f
paper_authors.name: Mojtaba Komeili
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:48:52.176Z
paper_authors._id: 659d717f2dbf5c8fee9c1fac
paper_authors.avatarUrl: /avatars/6bea1195499a66da9b48fe0c27781ba8.svg
paper_authors.fullname: Mojtaba Komeili
paper_authors.type: user
paper_authors.user: mkomeili
paper_authors._id: 64828cc35f581f4518496850
paper_authors.name: Eric Michael Smith
paper_authors._id: 64828cc35f581f4518496851
paper_authors.name: Megan Ung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-09T16:53:55.559Z
paper_authors._id: 63c6f88850cc81901da905f3
paper_authors.avatarUrl: /avatars/e75bb87af183aa5c3c5a7d09f2de1d60.svg
paper_authors.fullname: Megan Ung
paper_authors.type: user
paper_authors.user: meganung
paper_authors._id: 64828cc35f581f4518496852
paper_authors.name: Morteza Behrooz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:46:40.806Z
paper_authors._id: 64597ca038d95f3427a478e2
paper_authors.avatarUrl: /avatars/124de43a636728193e549460aa0f800c.svg
paper_authors.fullname: Morteza Behrooz
paper_authors.type: user
paper_authors.user: lessteza
paper_authors._id: 64828cc35f581f4518496853
paper_authors.name: William Ngan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:47:00.053Z
paper_authors._id: 630414995c70c21d0ead52b0
paper_authors.avatarUrl: /avatars/33cd9046d9833f0a9bc945ae776778e6.svg
paper_authors.fullname: William Ngan
paper_authors.type: user
paper_authors.user: metaphorical
paper_authors._id: 64828cc35f581f4518496854
paper_authors.name: Rashel Moritz
paper_authors._id: 64828cc35f581f4518496855
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T16:01:37.228Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_authors._id: 64828cc35f581f4518496856
paper_authors.name: Y-Lan Boureau
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-10T09:25:19.383Z
paper_authors._id: 64a82a24a1cd4978d49aa7cb
paper_authors.avatarUrl: /avatars/d7d7c3ad870541017551594bd30b6fde.svg
paper_authors.fullname: Y-Lan Boureau
paper_authors.type: user
paper_authors.user: notebleue
paper_authors._id: 64828cc35f581f4518496857
paper_authors.name: Jason Weston
paper_authors._id: 64828cc35f581f4518496858
paper_authors.name: Kurt Shuster
paper_title: Improving Open Language Models by Learning from Organic Interactions
paper_summary: We present BlenderBot 3x, an update on the conversational model BlenderBot 3,
  which is now trained using organic conversation and feedback data from
  participating users of the system in order to improve both its skills and
  safety. We are publicly releasing the participating de-identified interaction
  data for use by the research community, in order to spur further progress.
  Training models with organic data is challenging because interactions with
  people "in the wild" include both high quality conversations and feedback, as
  well as adversarial and toxic behavior. We study techniques that enable
  learning from helpful teachers while avoiding learning from people who are
  trying to trick the model into unhelpful or toxic responses. BlenderBot 3x is
  both preferred in conversation to BlenderBot 3, and is shown to produce safer
  responses in challenging situations. While our current models are still far
  from perfect, we believe further improvement can be achieved by continued use
  of the techniques explored in this work.
paper_ai_keywords: conversational model BlenderBot 3x BlenderBot 3 organic conversation feedback data de-identified text data research community adversarial behavior toxic behavior helpful teachers unhelpful responses toxic responses preferred challenging situations