title: GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning
    LLMs
summary: Robotic simulation today remains challenging to scale up due to the human
  efforts required to create diverse simulation tasks and scenes.
  Simulation-trained policies also face scalability issues as many sim-to-real
  methods focus on a single task. To address these challenges, this work proposes
  GenSim2, a scalable framework that leverages coding LLMs with multi-modal and
  reasoning capabilities for complex and realistic simulation task creation,
  including long-horizon tasks with articulated objects. To automatically
  generate demonstration data for these tasks at scale, we propose planning and
  RL solvers that generalize within object categories. The pipeline can generate
  data for up to 100 articulated tasks with 200 objects and reduce the required
  human efforts. To utilize such data, we propose an effective multi-task
  language-conditioned policy architecture, dubbed proprioceptive point-cloud
  transformer (PPT), that learns from the generated demonstrations and exhibits
  strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the
  policy architecture, we show a promising usage of GenSim2 that the generated
  data can be used for zero-shot transfer or co-train with real-world collected
  data, which enhances the policy performance by 20% compared with training
  exclusively on limited real data.
paper_authors._id: 6703efb3fa3ab087c58e6b87
paper_authors.name: Pu Hua
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T15:10:16.536Z
paper_authors._id: 6456445e1ca9debab051b5d0
paper_authors.avatarUrl: /avatars/769496beded27fa85957e62fa0b46313.svg
paper_authors.fullname: Pu Hua
paper_authors.type: user
paper_authors.user: piao-0429
paper_authors._id: 6703efb3fa3ab087c58e6b88
paper_authors.name: Minghuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T17:45:22.716Z
paper_authors._id: 630c8f0c8e3ff0c72327140a
paper_authors.avatarUrl: /avatars/998de36d102b8bec85624f209272f7f2.svg
paper_authors.fullname: Minghuan Liu
paper_authors.type: user
paper_authors.user: ericonaldo
paper_authors._id: 6703efb3fa3ab087c58e6b89
paper_authors.name: Annabella Macaluso
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T17:45:57.064Z
paper_authors._id: 65179a79db386e10222412ec
paper_authors.avatarUrl: /avatars/d25380d37221c26d131a14da76f3a04f.svg
paper_authors.fullname: Annabella Macaluso
paper_authors.type: user
paper_authors.user: amacalus
paper_authors._id: 6703efb3fa3ab087c58e6b8a
paper_authors.name: Yunfeng Lin
paper_authors._id: 6703efb3fa3ab087c58e6b8b
paper_authors.name: Weinan Zhang
paper_authors._id: 6703efb3fa3ab087c58e6b8c
paper_authors.name: Huazhe Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T17:46:31.354Z
paper_authors._id: 649c130f92525b3ac9fbcffe
paper_authors.avatarUrl: /avatars/b60e4d6803f7e4ef42200ddcb9e5bac1.svg
paper_authors.fullname: Huazhe Xu
paper_authors.type: user
paper_authors.user: Huazhe-Xu
paper_authors._id: 6703efb3fa3ab087c58e6b8d
paper_authors.name: Lirui Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T17:46:42.784Z
paper_authors._id: 63151385b031f7b1c7c0871c
paper_authors.avatarUrl: /avatars/0088eb929866face5f95218943e3f478.svg
paper_authors.fullname: Lirui Wang
paper_authors.type: user
paper_authors.user: liruiw
paper_title: GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning
    LLMs
paper_summary: Robotic simulation today remains challenging to scale up due to the human
  efforts required to create diverse simulation tasks and scenes.
  Simulation-trained policies also face scalability issues as many sim-to-real
  methods focus on a single task. To address these challenges, this work proposes
  GenSim2, a scalable framework that leverages coding LLMs with multi-modal and
  reasoning capabilities for complex and realistic simulation task creation,
  including long-horizon tasks with articulated objects. To automatically
  generate demonstration data for these tasks at scale, we propose planning and
  RL solvers that generalize within object categories. The pipeline can generate
  data for up to 100 articulated tasks with 200 objects and reduce the required
  human efforts. To utilize such data, we propose an effective multi-task
  language-conditioned policy architecture, dubbed proprioceptive point-cloud
  transformer (PPT), that learns from the generated demonstrations and exhibits
  strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the
  policy architecture, we show a promising usage of GenSim2 that the generated
  data can be used for zero-shot transfer or co-train with real-world collected
  data, which enhances the policy performance by 20% compared with training
  exclusively on limited real data.
paper_ai_summary: GenSim2 framework uses coding LLMs and RL solvers to generate complex simulation tasks and data, enabling a multi-task policy (PPT) with strong sim-to-real transfer performance.
paper_ai_keywords: coding LLMs multi-modal reasoning capabilities long-horizon tasks articulated objects planning RL solvers demonstration data multi-task language-conditioned policy architecture proprioceptive point-cloud transformer zero-shot transfer co-train