title: Policy Improvement using Language Feedback Models
summary: We introduce Language Feedback Models (LFMs) that identify desirable
  behaviour - actions that help achieve tasks specified in the instruction - for
  imitation learning in instruction following. To train LFMs, we obtain feedback
  from Large Language Models (LLMs) on visual trajectories verbalized to language
  descriptions. First, by using LFMs to identify desirable behaviour to imitate,
  we improve in task-completion rate over strong behavioural cloning baselines on
  three distinct language grounding environments (Touchdown, ScienceWorld, and
  ALFWorld). Second, LFMs outperform using LLMs as experts to directly predict
  actions, when controlling for the number of LLM output tokens. Third, LFMs
  generalize to unseen environments, improving task-completion rate by 3.5-12.0%
  through one round of adaptation. Finally, LFM can be modified to provide
  human-interpretable feedback without performance loss, allowing human
  verification of desirable behaviour for imitation learning.
paper_authors._id: 65cae68b5495933ab08201e0
paper_authors.name: Victor Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T08:22:34.814Z
paper_authors._id: 64b7f8ad324038715956501e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b7f8ad324038715956501e/FS8mWUV4zk4r2TsQuDFpE.jpeg
paper_authors.fullname: Victor Zhong
paper_authors.type: user
paper_authors.user: vzhong
paper_authors._id: 65cae68b5495933ab08201e1
paper_authors.name: Dipendra Misra
paper_authors._id: 65cae68b5495933ab08201e2
paper_authors.name: Xingdi Yuan
paper_authors._id: 65cae68b5495933ab08201e3
paper_authors.name: Marc-Alexandre Côté
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T08:23:13.012Z
paper_authors._id: 62960afb91760de5e547e544
paper_authors.avatarUrl: /avatars/bebed5926b04d066d4c0e49d246acd78.svg
paper_authors.fullname: Marc-Alexandre Côté
paper_authors.type: user
paper_authors.user: MarcCote
paper_title: Policy Improvement using Language Feedback Models
paper_summary: We introduce Language Feedback Models (LFMs) that identify desirable
  behaviour - actions that help achieve tasks specified in the instruction - for
  imitation learning in instruction following. To train LFMs, we obtain feedback
  from Large Language Models (LLMs) on visual trajectories verbalized to language
  descriptions. First, by using LFMs to identify desirable behaviour to imitate,
  we improve in task-completion rate over strong behavioural cloning baselines on
  three distinct language grounding environments (Touchdown, ScienceWorld, and
  ALFWorld). Second, LFMs outperform using LLMs as experts to directly predict
  actions, when controlling for the number of LLM output tokens. Third, LFMs
  generalize to unseen environments, improving task-completion rate by 3.5-12.0%
  through one round of adaptation. Finally, LFM can be modified to provide
  human-interpretable feedback without performance loss, allowing human
  verification of desirable behaviour for imitation learning.
paper_ai_keywords: Language Feedback Models (LFMs) imitation learning instruction following desirable behaviour adversarial feedback Large Language Models (LLMs) visual trajectories language descriptions behavioural cloning Touchdown ScienceWorld ALFWorld task-completion rate action prediction human-interpretable feedback