title: Goal Representations for Instruction Following: A Semi-Supervised
    Language Interface to Control
summary: Our goal is for robots to follow natural language instructions like "put the
  towel next to the microwave." But getting large amounts of labeled data, i.e.
  data that contains demonstrations of tasks labeled with the language
  instruction, is prohibitive. In contrast, obtaining policies that respond to
  image goals is much easier, because any autonomous trial or demonstration can
  be labeled in hindsight with its final state as the goal. In this work, we
  contribute a method that taps into joint image- and goal- conditioned policies
  with language using only a small amount of language data. Prior work has made
  progress on this using vision-language models or by jointly training
  language-goal-conditioned policies, but so far neither method has scaled
  effectively to real-world robot tasks without significant human annotation. Our
  method achieves robust performance in the real world by learning an embedding
  from the labeled data that aligns language not to the goal image, but rather to
  the desired change between the start and goal images that the instruction
  corresponds to. We then train a policy on this embedding: the policy benefits
  from all the unlabeled data, but the aligned embedding provides an interface
  for language to steer the policy. We show instruction following across a
  variety of manipulation tasks in different scenes, with generalization to
  language instructions outside of the labeled data. Videos and code for our
  approach can be found on our website: http://tiny.cc/grif .
paper_authors._id: 64a37def6badc2ebd1dde7be
paper_authors.name: Vivek Myers
paper_authors._id: 64a37def6badc2ebd1dde7bf
paper_authors.name: Andre He
paper_authors._id: 64a37def6badc2ebd1dde7c0
paper_authors.name: Kuan Fang
paper_authors._id: 64a37def6badc2ebd1dde7c1
paper_authors.name: Homer Walke
paper_authors._id: 64a37def6badc2ebd1dde7c2
paper_authors.name: Philippe Hansen-Estruch
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:05:49.347Z
paper_authors._id: 644eb923a8f384abd038a4f7
paper_authors.avatarUrl: /avatars/3d1a29c6149a6ffc482842f1438ba55a.svg
paper_authors.fullname: Philippe I Hansen-Estruch
paper_authors.type: user
paper_authors.user: philippehansen
paper_authors._id: 64a37def6badc2ebd1dde7c3
paper_authors.name: Ching-An Cheng
paper_authors._id: 64a37def6badc2ebd1dde7c4
paper_authors.name: Mihai Jalobeanu
paper_authors._id: 64a37def6badc2ebd1dde7c5
paper_authors.name: Andrey Kolobov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-04T19:17:12.661Z
paper_authors._id: 6350478dc5acdbefb8ecf722
paper_authors.avatarUrl: /avatars/75f3cdc38f6e67ae0435c30aff50b089.svg
paper_authors.fullname: Andrey Kolobov
paper_authors.type: user
paper_authors.user: akolobov
paper_authors._id: 64a37def6badc2ebd1dde7c6
paper_authors.name: Anca Dragan
paper_authors._id: 64a37def6badc2ebd1dde7c7
paper_authors.name: Sergey Levine
paper_title: Goal Representations for Instruction Following: A Semi-Supervised
    Language Interface to Control
paper_summary: Our goal is for robots to follow natural language instructions like "put the
  towel next to the microwave." But getting large amounts of labeled data, i.e.
  data that contains demonstrations of tasks labeled with the language
  instruction, is prohibitive. In contrast, obtaining policies that respond to
  image goals is much easier, because any autonomous trial or demonstration can
  be labeled in hindsight with its final state as the goal. In this work, we
  contribute a method that taps into joint image- and goal- conditioned policies
  with language using only a small amount of language data. Prior work has made
  progress on this using vision-language models or by jointly training
  language-goal-conditioned policies, but so far neither method has scaled
  effectively to real-world robot tasks without significant human annotation. Our
  method achieves robust performance in the real world by learning an embedding
  from the labeled data that aligns language not to the goal image, but rather to
  the desired change between the start and goal images that the instruction
  corresponds to. We then train a policy on this embedding: the policy benefits
  from all the unlabeled data, but the aligned embedding provides an interface
  for language to steer the policy. We show instruction following across a
  variety of manipulation tasks in different scenes, with generalization to
  language instructions outside of the labeled data. Videos and code for our
  approach can be found on our website: http://tiny.cc/grif .
paper_ai_summary: A method is introduced to enable robots to follow natural language instructions with minimal labeled data by aligning language to the change between images rather than the goal image, enhancing policy training and generalization.
paper_ai_keywords: vision-language models language-goal-conditioned policies embedding instruction following manipulation tasks