RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches
Natural language and images are commonly used as goal representations in
goal-conditioned imitation learning (IL). However, natural language can be
ambiguous and images can be over-specified. In this work, we propose hand-drawn
sketches as a modality for goal specification in visual imitation learning.
Sketches are easy for users to provide on the fly like language, but similar to
images they can also help a downstream policy to be spatially-aware and even go
beyond images to disambiguate task-relevant from task-irrelevant objects. We
present RT-Sketch, a goal-conditioned policy for manipulation that takes a
hand-drawn sketch of the desired scene as input, and outputs actions. We train
RT-Sketch on a dataset of paired trajectories and corresponding synthetically
generated goal sketches. We evaluate this approach on six manipulation skills
involving tabletop object rearrangements on an articulated countertop.
Experimentally we find that RT-Sketch is able to perform on a similar level to
image or language-conditioned agents in straightforward settings, while
achieving greater robustness when language goals are ambiguous or visual
distractors are present. Additionally, we show that RT-Sketch has the capacity
to interpret and act upon sketches with varied levels of specificity, ranging
from minimal line drawings to detailed, colored drawings. For supplementary
material and videos, please refer to our website: http://rt-sketch.github.io.
65e7fcc8e3ef3be5fdf8d5fd
Priya Sundaresan
admin_assigned
2024-03-06T10:04:35.137Z
633db7f6615cbd4fda734197
/avatars/23ce83ab2076f0edb7a89143ad87f35a.svg
Priya Sundaresan
user
priyasun
65e7fcc8e3ef3be5fdf8d5fe
Quan Vuong
admin_assigned
2024-03-06T10:04:43.250Z
63e1d6687fbb6ae4d4f4d980
/avatars/4959319e69d69e03a31ff4be326e602a.svg
Quan Vuong
user
quintonx
65e7fcc8e3ef3be5fdf8d5ff
Jiayuan Gu
admin_assigned
2024-03-06T10:04:49.720Z
638aa283cebef0d13aa2ec2e
/avatars/3b67e6a6073033864a817230e97c27ca.svg
Jiayuan Gu
user
jigu
65e7fcc8e3ef3be5fdf8d600
Peng Xu
65e7fcc8e3ef3be5fdf8d601
Ted Xiao
65e7fcc8e3ef3be5fdf8d602
Sean Kirmani
admin_assigned
2024-03-06T10:05:31.025Z
648c85f33ffc11989bddcce1
/avatars/bdde73b95a36e32cb2975656cca46022.svg
Sean Kirmani
user
skirmani
65e7fcc8e3ef3be5fdf8d603
Tianhe Yu
admin_assigned
2024-03-06T10:05:38.354Z
64c95527d33e6a7f82b8b911
/avatars/5efcf5d5a84766bbd3a5ec2354233f1c.svg
Tianhe Yu
user
keviny66
65e7fcc8e3ef3be5fdf8d604
Michael Stark
65e7fcc8e3ef3be5fdf8d605
Ajinkya Jain
65e7fcc8e3ef3be5fdf8d606
Karol Hausman
admin_assigned
2024-03-06T10:07:35.303Z
64c93e623a3f428da67d9310
/avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
Karol Hausman
user
KarolHausman
65e7fcc8e3ef3be5fdf8d607
Dorsa Sadigh
65e7fcc8e3ef3be5fdf8d608
Jeannette Bohg
claimed_verified
2024-12-18T09:55:15.523Z
676272a59865895e253ceee5
https://cdn-avatars.huggingface.co/v1/production/uploads/676272a59865895e253ceee5/Nz51Y-G49SD19lQEjH6Yd.jpeg
Jeannette Bohg
user
jbohg
65e7fcc8e3ef3be5fdf8d609
Stefan Schaal
RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches
Natural language and images are commonly used as goal representations in
goal-conditioned imitation learning (IL). However, natural language can be
ambiguous and images can be over-specified. In this work, we propose hand-drawn
sketches as a modality for goal specification in visual imitation learning.
Sketches are easy for users to provide on the fly like language, but similar to
images they can also help a downstream policy to be spatially-aware and even go
beyond images to disambiguate task-relevant from task-irrelevant objects. We
present RT-Sketch, a goal-conditioned policy for manipulation that takes a
hand-drawn sketch of the desired scene as input, and outputs actions. We train
RT-Sketch on a dataset of paired trajectories and corresponding synthetically
generated goal sketches. We evaluate this approach on six manipulation skills
involving tabletop object rearrangements on an articulated countertop.
Experimentally we find that RT-Sketch is able to perform on a similar level to
image or language-conditioned agents in straightforward settings, while
achieving greater robustness when language goals are ambiguous or visual
distractors are present. Additionally, we show that RT-Sketch has the capacity
to interpret and act upon sketches with varied levels of specificity, ranging
from minimal line drawings to detailed, colored drawings. For supplementary
material and videos, please refer to our website: http://rt-sketch.github.io.
RT-Sketch, a goal-conditioned policy using hand-drawn sketches for manipulation, demonstrates robustness and adaptability in varied settings over language and image-conditioned agents.
goal-conditioned imitation learning visual imitation learning hand-drawn sketches manipulation policy goal specification spatial-awareness task-relevant objects goal-sketches paired trajectories object rearrangements articulated countertop manipulation skills ambiguous language goals visual distractors