RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse
  Tools
Humans excel in complex long-horizon soft body manipulation tasks via
flexible tool use: bread baking requires a knife to slice the dough and a
rolling pin to flatten it. Often regarded as a hallmark of human cognition,
tool use in autonomous robots remains limited due to challenges in
understanding tool-object interactions. Here we develop an intelligent robotic
system, RoboCook, which perceives, models, and manipulates elasto-plastic
objects with various tools. RoboCook uses point cloud scene representations,
models tool-object interactions with Graph Neural Networks (GNNs), and combines
tool classification with self-supervised policy learning to devise manipulation
plans. We demonstrate that from just 20 minutes of real-world interaction data
per tool, a general-purpose robot arm can learn complex long-horizon soft
object manipulation tasks, such as making dumplings and alphabet letter
cookies. Extensive evaluations show that RoboCook substantially outperforms
state-of-the-art approaches, exhibits robustness against severe external
disturbances, and demonstrates adaptability to different materials.
649a887a685215f7248a7602
Haochen Shi
claimed_verified
2023-06-28T20:15:33.394Z
649c6f724cc1419329386141
/avatars/10efdd9a7cd4200a021bb342cd5f2481.svg
Haochen Shi
user
hshi74
649a887a685215f7248a7603
Huazhe Xu
claimed_verified
2023-06-28T20:15:26.624Z
649c130f92525b3ac9fbcffe
/avatars/b60e4d6803f7e4ef42200ddcb9e5bac1.svg
Huazhe Xu
user
Huazhe-Xu
649a887a685215f7248a7604
Samuel Clarke
649a887a685215f7248a7605
Yunzhu Li
claimed_verified
2023-07-04T19:17:32.373Z
64a46562c641afb468a3dc04
/avatars/bc54e5c13f657251b05bf2725086e56d.svg
Yunzhu Li
user
yunzhuli
649a887a685215f7248a7606
Jiajun Wu
claimed_verified
2023-06-28T20:15:29.369Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse
  Tools
Humans excel in complex long-horizon soft body manipulation tasks via
flexible tool use: bread baking requires a knife to slice the dough and a
rolling pin to flatten it. Often regarded as a hallmark of human cognition,
tool use in autonomous robots remains limited due to challenges in
understanding tool-object interactions. Here we develop an intelligent robotic
system, RoboCook, which perceives, models, and manipulates elasto-plastic
objects with various tools. RoboCook uses point cloud scene representations,
models tool-object interactions with Graph Neural Networks (GNNs), and combines
tool classification with self-supervised policy learning to devise manipulation
plans. We demonstrate that from just 20 minutes of real-world interaction data
per tool, a general-purpose robot arm can learn complex long-horizon soft
object manipulation tasks, such as making dumplings and alphabet letter
cookies. Extensive evaluations show that RoboCook substantially outperforms
state-of-the-art approaches, exhibits robustness against severe external
disturbances, and demonstrates adaptability to different materials.
A robotic system, RoboCook, utilizes point cloud scene representations and Graph Neural Networks to learn and adapt to complex soft object manipulation tasks with flexible tools, outperforming existing methods.
point cloud scene representations Graph Neural Networks (GNNs) tool-object interactions self-supervised policy learning long-horizon soft object manipulation general-purpose robot arm dumplings alphabet letter cookies robustness adaptability