title: RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches
summary: Natural language and images are commonly used as goal representations in
  goal-conditioned imitation learning (IL). However, natural language can be
  ambiguous and images can be over-specified. In this work, we propose hand-drawn
  sketches as a modality for goal specification in visual imitation learning.
  Sketches are easy for users to provide on the fly like language, but similar to
  images they can also help a downstream policy to be spatially-aware and even go
  beyond images to disambiguate task-relevant from task-irrelevant objects. We
  present RT-Sketch, a goal-conditioned policy for manipulation that takes a
  hand-drawn sketch of the desired scene as input, and outputs actions. We train
  RT-Sketch on a dataset of paired trajectories and corresponding synthetically
  generated goal sketches. We evaluate this approach on six manipulation skills
  involving tabletop object rearrangements on an articulated countertop.
  Experimentally we find that RT-Sketch is able to perform on a similar level to
  image or language-conditioned agents in straightforward settings, while
  achieving greater robustness when language goals are ambiguous or visual
  distractors are present. Additionally, we show that RT-Sketch has the capacity
  to interpret and act upon sketches with varied levels of specificity, ranging
  from minimal line drawings to detailed, colored drawings. For supplementary
  material and videos, please refer to our website: http://rt-sketch.github.io.
paper_authors._id: 65e7fcc8e3ef3be5fdf8d5fd
paper_authors.name: Priya Sundaresan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:04:35.137Z
paper_authors._id: 633db7f6615cbd4fda734197
paper_authors.avatarUrl: /avatars/23ce83ab2076f0edb7a89143ad87f35a.svg
paper_authors.fullname: Priya Sundaresan
paper_authors.type: user
paper_authors.user: priyasun
paper_authors._id: 65e7fcc8e3ef3be5fdf8d5fe
paper_authors.name: Quan Vuong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:04:43.250Z
paper_authors._id: 63e1d6687fbb6ae4d4f4d980
paper_authors.avatarUrl: /avatars/4959319e69d69e03a31ff4be326e602a.svg
paper_authors.fullname: Quan Vuong
paper_authors.type: user
paper_authors.user: quintonx
paper_authors._id: 65e7fcc8e3ef3be5fdf8d5ff
paper_authors.name: Jiayuan Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:04:49.720Z
paper_authors._id: 638aa283cebef0d13aa2ec2e
paper_authors.avatarUrl: /avatars/3b67e6a6073033864a817230e97c27ca.svg
paper_authors.fullname: Jiayuan Gu
paper_authors.type: user
paper_authors.user: jigu
paper_authors._id: 65e7fcc8e3ef3be5fdf8d600
paper_authors.name: Peng Xu
paper_authors._id: 65e7fcc8e3ef3be5fdf8d601
paper_authors.name: Ted Xiao
paper_authors._id: 65e7fcc8e3ef3be5fdf8d602
paper_authors.name: Sean Kirmani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:05:31.025Z
paper_authors._id: 648c85f33ffc11989bddcce1
paper_authors.avatarUrl: /avatars/bdde73b95a36e32cb2975656cca46022.svg
paper_authors.fullname: Sean Kirmani
paper_authors.type: user
paper_authors.user: skirmani
paper_authors._id: 65e7fcc8e3ef3be5fdf8d603
paper_authors.name: Tianhe Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:05:38.354Z
paper_authors._id: 64c95527d33e6a7f82b8b911
paper_authors.avatarUrl: /avatars/5efcf5d5a84766bbd3a5ec2354233f1c.svg
paper_authors.fullname: Tianhe Yu
paper_authors.type: user
paper_authors.user: keviny66
paper_authors._id: 65e7fcc8e3ef3be5fdf8d604
paper_authors.name: Michael Stark
paper_authors._id: 65e7fcc8e3ef3be5fdf8d605
paper_authors.name: Ajinkya Jain
paper_authors._id: 65e7fcc8e3ef3be5fdf8d606
paper_authors.name: Karol Hausman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:07:35.303Z
paper_authors._id: 64c93e623a3f428da67d9310
paper_authors.avatarUrl: /avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
paper_authors.fullname: Karol Hausman
paper_authors.type: user
paper_authors.user: KarolHausman
paper_authors._id: 65e7fcc8e3ef3be5fdf8d607
paper_authors.name: Dorsa Sadigh
paper_authors._id: 65e7fcc8e3ef3be5fdf8d608
paper_authors.name: Jeannette Bohg
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T09:55:15.523Z
paper_authors._id: 676272a59865895e253ceee5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/676272a59865895e253ceee5/Nz51Y-G49SD19lQEjH6Yd.jpeg
paper_authors.fullname: Jeannette Bohg
paper_authors.type: user
paper_authors.user: jbohg
paper_authors._id: 65e7fcc8e3ef3be5fdf8d609
paper_authors.name: Stefan Schaal
paper_title: RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches
paper_summary: Natural language and images are commonly used as goal representations in
  goal-conditioned imitation learning (IL). However, natural language can be
  ambiguous and images can be over-specified. In this work, we propose hand-drawn
  sketches as a modality for goal specification in visual imitation learning.
  Sketches are easy for users to provide on the fly like language, but similar to
  images they can also help a downstream policy to be spatially-aware and even go
  beyond images to disambiguate task-relevant from task-irrelevant objects. We
  present RT-Sketch, a goal-conditioned policy for manipulation that takes a
  hand-drawn sketch of the desired scene as input, and outputs actions. We train
  RT-Sketch on a dataset of paired trajectories and corresponding synthetically
  generated goal sketches. We evaluate this approach on six manipulation skills
  involving tabletop object rearrangements on an articulated countertop.
  Experimentally we find that RT-Sketch is able to perform on a similar level to
  image or language-conditioned agents in straightforward settings, while
  achieving greater robustness when language goals are ambiguous or visual
  distractors are present. Additionally, we show that RT-Sketch has the capacity
  to interpret and act upon sketches with varied levels of specificity, ranging
  from minimal line drawings to detailed, colored drawings. For supplementary
  material and videos, please refer to our website: http://rt-sketch.github.io.
paper_ai_summary: RT-Sketch, a goal-conditioned policy using hand-drawn sketches for manipulation, demonstrates robustness and adaptability in varied settings over language and image-conditioned agents.
paper_ai_keywords: goal-conditioned imitation learning visual imitation learning hand-drawn sketches manipulation policy goal specification spatial-awareness task-relevant objects goal-sketches paired trajectories object rearrangements articulated countertop manipulation skills ambiguous language goals visual distractors