GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning
  LLMs
Robotic simulation today remains challenging to scale up due to the human
efforts required to create diverse simulation tasks and scenes.
Simulation-trained policies also face scalability issues as many sim-to-real
methods focus on a single task. To address these challenges, this work proposes
GenSim2, a scalable framework that leverages coding LLMs with multi-modal and
reasoning capabilities for complex and realistic simulation task creation,
including long-horizon tasks with articulated objects. To automatically
generate demonstration data for these tasks at scale, we propose planning and
RL solvers that generalize within object categories. The pipeline can generate
data for up to 100 articulated tasks with 200 objects and reduce the required
human efforts. To utilize such data, we propose an effective multi-task
language-conditioned policy architecture, dubbed proprioceptive point-cloud
transformer (PPT), that learns from the generated demonstrations and exhibits
strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the
policy architecture, we show a promising usage of GenSim2 that the generated
data can be used for zero-shot transfer or co-train with real-world collected
data, which enhances the policy performance by 20% compared with training
exclusively on limited real data.
6703efb3fa3ab087c58e6b87
Pu Hua
claimed_verified
2024-10-07T15:10:16.536Z
6456445e1ca9debab051b5d0
/avatars/769496beded27fa85957e62fa0b46313.svg
Pu Hua
user
piao-0429
6703efb3fa3ab087c58e6b88
Minghuan Liu
admin_assigned
2024-10-07T17:45:22.716Z
630c8f0c8e3ff0c72327140a
/avatars/998de36d102b8bec85624f209272f7f2.svg
Minghuan Liu
user
ericonaldo
6703efb3fa3ab087c58e6b89
Annabella Macaluso
admin_assigned
2024-10-07T17:45:57.064Z
65179a79db386e10222412ec
/avatars/d25380d37221c26d131a14da76f3a04f.svg
Annabella Macaluso
user
amacalus
6703efb3fa3ab087c58e6b8a
Yunfeng Lin
6703efb3fa3ab087c58e6b8b
Weinan Zhang
6703efb3fa3ab087c58e6b8c
Huazhe Xu
admin_assigned
2024-10-07T17:46:31.354Z
649c130f92525b3ac9fbcffe
/avatars/b60e4d6803f7e4ef42200ddcb9e5bac1.svg
Huazhe Xu
user
Huazhe-Xu
6703efb3fa3ab087c58e6b8d
Lirui Wang
admin_assigned
2024-10-07T17:46:42.784Z
63151385b031f7b1c7c0871c
/avatars/0088eb929866face5f95218943e3f478.svg
Lirui Wang
user
liruiw
GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning
  LLMs
Robotic simulation today remains challenging to scale up due to the human
efforts required to create diverse simulation tasks and scenes.
Simulation-trained policies also face scalability issues as many sim-to-real
methods focus on a single task. To address these challenges, this work proposes
GenSim2, a scalable framework that leverages coding LLMs with multi-modal and
reasoning capabilities for complex and realistic simulation task creation,
including long-horizon tasks with articulated objects. To automatically
generate demonstration data for these tasks at scale, we propose planning and
RL solvers that generalize within object categories. The pipeline can generate
data for up to 100 articulated tasks with 200 objects and reduce the required
human efforts. To utilize such data, we propose an effective multi-task
language-conditioned policy architecture, dubbed proprioceptive point-cloud
transformer (PPT), that learns from the generated demonstrations and exhibits
strong sim-to-real zero-shot transfer. Combining the proposed pipeline and the
policy architecture, we show a promising usage of GenSim2 that the generated
data can be used for zero-shot transfer or co-train with real-world collected
data, which enhances the policy performance by 20% compared with training
exclusively on limited real data.
GenSim2 framework uses coding LLMs and RL solvers to generate complex simulation tasks and data, enabling a multi-task policy (PPT) with strong sim-to-real transfer performance.
coding LLMs multi-modal reasoning capabilities long-horizon tasks articulated objects planning RL solvers demonstration data multi-task language-conditioned policy architecture proprioceptive point-cloud transformer zero-shot transfer co-train