title: RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse
    Tools
summary: Humans excel in complex long-horizon soft body manipulation tasks via
  flexible tool use: bread baking requires a knife to slice the dough and a
  rolling pin to flatten it. Often regarded as a hallmark of human cognition,
  tool use in autonomous robots remains limited due to challenges in
  understanding tool-object interactions. Here we develop an intelligent robotic
  system, RoboCook, which perceives, models, and manipulates elasto-plastic
  objects with various tools. RoboCook uses point cloud scene representations,
  models tool-object interactions with Graph Neural Networks (GNNs), and combines
  tool classification with self-supervised policy learning to devise manipulation
  plans. We demonstrate that from just 20 minutes of real-world interaction data
  per tool, a general-purpose robot arm can learn complex long-horizon soft
  object manipulation tasks, such as making dumplings and alphabet letter
  cookies. Extensive evaluations show that RoboCook substantially outperforms
  state-of-the-art approaches, exhibits robustness against severe external
  disturbances, and demonstrates adaptability to different materials.
paper_authors._id: 649a887a685215f7248a7602
paper_authors.name: Haochen Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:15:33.394Z
paper_authors._id: 649c6f724cc1419329386141
paper_authors.avatarUrl: /avatars/10efdd9a7cd4200a021bb342cd5f2481.svg
paper_authors.fullname: Haochen Shi
paper_authors.type: user
paper_authors.user: hshi74
paper_authors._id: 649a887a685215f7248a7603
paper_authors.name: Huazhe Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:15:26.624Z
paper_authors._id: 649c130f92525b3ac9fbcffe
paper_authors.avatarUrl: /avatars/b60e4d6803f7e4ef42200ddcb9e5bac1.svg
paper_authors.fullname: Huazhe Xu
paper_authors.type: user
paper_authors.user: Huazhe-Xu
paper_authors._id: 649a887a685215f7248a7604
paper_authors.name: Samuel Clarke
paper_authors._id: 649a887a685215f7248a7605
paper_authors.name: Yunzhu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-04T19:17:32.373Z
paper_authors._id: 64a46562c641afb468a3dc04
paper_authors.avatarUrl: /avatars/bc54e5c13f657251b05bf2725086e56d.svg
paper_authors.fullname: Yunzhu Li
paper_authors.type: user
paper_authors.user: yunzhuli
paper_authors._id: 649a887a685215f7248a7606
paper_authors.name: Jiajun Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:15:29.369Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_title: RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse
    Tools
paper_summary: Humans excel in complex long-horizon soft body manipulation tasks via
  flexible tool use: bread baking requires a knife to slice the dough and a
  rolling pin to flatten it. Often regarded as a hallmark of human cognition,
  tool use in autonomous robots remains limited due to challenges in
  understanding tool-object interactions. Here we develop an intelligent robotic
  system, RoboCook, which perceives, models, and manipulates elasto-plastic
  objects with various tools. RoboCook uses point cloud scene representations,
  models tool-object interactions with Graph Neural Networks (GNNs), and combines
  tool classification with self-supervised policy learning to devise manipulation
  plans. We demonstrate that from just 20 minutes of real-world interaction data
  per tool, a general-purpose robot arm can learn complex long-horizon soft
  object manipulation tasks, such as making dumplings and alphabet letter
  cookies. Extensive evaluations show that RoboCook substantially outperforms
  state-of-the-art approaches, exhibits robustness against severe external
  disturbances, and demonstrates adaptability to different materials.
paper_ai_keywords: point cloud scene representations Graph Neural Networks (GNNs) tool classification self-supervised policy learning elasto-plastic objects RoboCook Dumplings alphabet letter cookies