title: Retrospective Learning from Interactions
summary: Multi-turn interactions between large language models (LLMs) and users
  naturally include implicit feedback signals. If an LLM responds in an
  unexpected way to an instruction, the user is likely to signal it by rephrasing
  the request, expressing frustration, or pivoting to an alternative task. Such
  signals are task-independent and occupy a relatively constrained subspace of
  language, allowing the LLM to identify them even if it fails on the actual
  task. This creates an avenue for continually learning from interactions without
  additional annotations. We introduce ReSpect, a method to learn from such
  signals in past interactions via retrospection. We deploy ReSpect in a new
  multimodal interaction scenario, where humans instruct an LLM to solve an
  abstract reasoning task with a combinatorial solution space. Through thousands
  of interactions with humans, we show how ReSpect gradually improves task
  completion rate from 31% to 82%, all without any external annotation.
paper_authors._id: 6711c3ba7da5c0a86f61f0c5
paper_authors.name: Zizhao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:54:54.582Z
paper_authors._id: 634642c17393804ce0d83879
paper_authors.avatarUrl: /avatars/36ef4df22b9faaf76a8a03e2c55a66e8.svg
paper_authors.fullname: zizhao chen
paper_authors.type: user
paper_authors.user: czz
paper_authors._id: 6711c3ba7da5c0a86f61f0c6
paper_authors.name: Mustafa Omer Gul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:55:00.071Z
paper_authors._id: 640261cc06c715b934025f4e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640261cc06c715b934025f4e/aXnh-ZGvEahSovuivi6dr.jpeg
paper_authors.fullname: Mustafa Omer Gul
paper_authors.type: user
paper_authors.user: momergul
paper_authors._id: 6711c3ba7da5c0a86f61f0c7
paper_authors.name: Yiwei Chen
paper_authors._id: 6711c3ba7da5c0a86f61f0c8
paper_authors.name: Gloria Geng
paper_authors._id: 6711c3ba7da5c0a86f61f0c9
paper_authors.name: Anne Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:55:59.916Z
paper_authors._id: 64515bda110343638e38b3df
paper_authors.avatarUrl: /avatars/a69ccb8cc40a58d49fa907a00a1397fd.svg
paper_authors.fullname: Anne Wu
paper_authors.type: user
paper_authors.user: anneyouw
paper_authors._id: 6711c3ba7da5c0a86f61f0ca
paper_authors.name: Yoav Artzi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:56:05.282Z
paper_authors._id: 637c23489495870ef76d7880
paper_authors.avatarUrl: /avatars/4fd2f23d9a487b1ec0c749c1e11b8ffc.svg
paper_authors.fullname: Yoav Artzi
paper_authors.type: user
paper_authors.user: yoavartzi
paper_title: Retrospective Learning from Interactions
paper_summary: Multi-turn interactions between large language models (LLMs) and users
  naturally include implicit feedback signals. If an LLM responds in an
  unexpected way to an instruction, the user is likely to signal it by rephrasing
  the request, expressing frustration, or pivoting to an alternative task. Such
  signals are task-independent and occupy a relatively constrained subspace of
  language, allowing the LLM to identify them even if it fails on the actual
  task. This creates an avenue for continually learning from interactions without
  additional annotations. We introduce ReSpect, a method to learn from such
  signals in past interactions via retrospection. We deploy ReSpect in a new
  multimodal interaction scenario, where humans instruct an LLM to solve an
  abstract reasoning task with a combinatorial solution space. Through thousands
  of interactions with humans, we show how ReSpect gradually improves task
  completion rate from 31% to 82%, all without any external annotation.
paper_ai_keywords: large language models (LLMs) implicit feedback signals rephrasing frustration pivoting task-independent constrained subspace retrospection multimodal interaction scenario abstract reasoning task combinatorial solution space task completion rate