RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation
For robots to be useful outside labs and specialized factories we need a way
to teach them new useful behaviors quickly. Current approaches lack either the
generality to onboard new tasks without task-specific engineering, or else lack
the data-efficiency to do so in an amount of time that enables practical use.
In this work we explore dense tracking as a representational vehicle to allow
faster and more general learning from demonstration. Our approach utilizes
Track-Any-Point (TAP) models to isolate the relevant motion in a demonstration,
and parameterize a low-level controller to reproduce this motion across changes
in the scene configuration. We show this results in robust robot policies that
can solve complex object-arrangement tasks such as shape-matching, stacking,
and even full path-following tasks such as applying glue and sticking objects
together, all from demonstrations that can be collected in minutes.
64f00e7ac05f2d3c65a1ca52
Mel Vecerik
64f00e7ac05f2d3c65a1ca53
Carl Doersch
64f00e7ac05f2d3c65a1ca54
Yi Yang
64f00e7ac05f2d3c65a1ca55
Todor Davchev
64f00e7ac05f2d3c65a1ca56
Yusuf Aytar
64f00e7ac05f2d3c65a1ca57
Guangyao Zhou
64f00e7ac05f2d3c65a1ca58
Raia Hadsell
64f00e7ac05f2d3c65a1ca59
Lourdes Agapito
64f00e7ac05f2d3c65a1ca5a
Jon Scholz
RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation
For robots to be useful outside labs and specialized factories we need a way
to teach them new useful behaviors quickly. Current approaches lack either the
generality to onboard new tasks without task-specific engineering, or else lack
the data-efficiency to do so in an amount of time that enables practical use.
In this work we explore dense tracking as a representational vehicle to allow
faster and more general learning from demonstration. Our approach utilizes
Track-Any-Point (TAP) models to isolate the relevant motion in a demonstration,
and parameterize a low-level controller to reproduce this motion across changes
in the scene configuration. We show this results in robust robot policies that
can solve complex object-arrangement tasks such as shape-matching, stacking,
and even full path-following tasks such as applying glue and sticking objects
together, all from demonstrations that can be collected in minutes.
Robots can learn new tasks more quickly and generally using dense tracking and parameterized low-level controllers, as demonstrated with TAP models.
Track-Any-Point dense tracking low-level controller shape-matching stacking path-following