title: RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation
summary: For robots to be useful outside labs and specialized factories we need a way
  to teach them new useful behaviors quickly. Current approaches lack either the
  generality to onboard new tasks without task-specific engineering, or else lack
  the data-efficiency to do so in an amount of time that enables practical use.
  In this work we explore dense tracking as a representational vehicle to allow
  faster and more general learning from demonstration. Our approach utilizes
  Track-Any-Point (TAP) models to isolate the relevant motion in a demonstration,
  and parameterize a low-level controller to reproduce this motion across changes
  in the scene configuration. We show this results in robust robot policies that
  can solve complex object-arrangement tasks such as shape-matching, stacking,
  and even full path-following tasks such as applying glue and sticking objects
  together, all from demonstrations that can be collected in minutes.
paper_authors._id: 64f00e7ac05f2d3c65a1ca52
paper_authors.name: Mel Vecerik
paper_authors._id: 64f00e7ac05f2d3c65a1ca53
paper_authors.name: Carl Doersch
paper_authors._id: 64f00e7ac05f2d3c65a1ca54
paper_authors.name: Yi Yang
paper_authors._id: 64f00e7ac05f2d3c65a1ca55
paper_authors.name: Todor Davchev
paper_authors._id: 64f00e7ac05f2d3c65a1ca56
paper_authors.name: Yusuf Aytar
paper_authors._id: 64f00e7ac05f2d3c65a1ca57
paper_authors.name: Guangyao Zhou
paper_authors._id: 64f00e7ac05f2d3c65a1ca58
paper_authors.name: Raia Hadsell
paper_authors._id: 64f00e7ac05f2d3c65a1ca59
paper_authors.name: Lourdes Agapito
paper_authors._id: 64f00e7ac05f2d3c65a1ca5a
paper_authors.name: Jon Scholz
paper_title: RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation
paper_summary: For robots to be useful outside labs and specialized factories we need a way
  to teach them new useful behaviors quickly. Current approaches lack either the
  generality to onboard new tasks without task-specific engineering, or else lack
  the data-efficiency to do so in an amount of time that enables practical use.
  In this work we explore dense tracking as a representational vehicle to allow
  faster and more general learning from demonstration. Our approach utilizes
  Track-Any-Point (TAP) models to isolate the relevant motion in a demonstration,
  and parameterize a low-level controller to reproduce this motion across changes
  in the scene configuration. We show this results in robust robot policies that
  can solve complex object-arrangement tasks such as shape-matching, stacking,
  and even full path-following tasks such as applying glue and sticking objects
  together, all from demonstrations that can be collected in minutes.
paper_ai_summary: Robots can learn new tasks more quickly and generally using dense tracking and parameterized low-level controllers, as demonstrated with TAP models.
paper_ai_keywords: Track-Any-Point dense tracking low-level controller shape-matching stacking path-following