title: LEGENT: Open Platform for Embodied Agents
summary: Despite advancements in Large Language Models (LLMs) and Large Multimodal
  Models (LMMs), their integration into language-grounded, human-like embodied
  agents remains incomplete, hindering complex real-life task performance in
  physical environments. Existing integrations often feature limited open
  sourcing, challenging collective progress in this field. We introduce LEGENT,
  an open, scalable platform for developing embodied agents using LLMs and LMMs.
  LEGENT offers a dual approach: a rich, interactive 3D environment with
  communicable and actionable agents, paired with a user-friendly interface, and
  a sophisticated data generation pipeline utilizing advanced algorithms to
  exploit supervision from simulated worlds at scale. In our experiments, an
  embryonic vision-language-action model trained on LEGENT-generated data
  surpasses GPT-4V in embodied tasks, showcasing promising generalization
  capabilities.
paper_authors._id: 663058fb33ccd9e68c4b47d7
paper_authors.name: Zhili Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:38:32.998Z
paper_authors._id: 635563ce525beaee688c4fc8
paper_authors.avatarUrl: /avatars/f74eec56da76dd77a23649b76b857d8d.svg
paper_authors.fullname: Zhili Cheng
paper_authors.type: user
paper_authors.user: chengzl18
paper_authors._id: 663058fb33ccd9e68c4b47d8
paper_authors.name: Zhitong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:38:44.960Z
paper_authors._id: 64265a0ba5ec4a5cbc532cb1
paper_authors.avatarUrl: /avatars/9c7fe642d510a3cb880b44ea8e129651.svg
paper_authors.fullname: Wang Zhitong
paper_authors.type: user
paper_authors.user: tommywang721
paper_authors._id: 663058fb33ccd9e68c4b47d9
paper_authors.name: Jinyi Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-30T07:27:38.900Z
paper_authors._id: 637a06580a77f602dc4ac922
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637a06580a77f602dc4ac922/JR2VHWhG1tj3TKKNzD8GD.jpeg
paper_authors.fullname: Jinyi Hu
paper_authors.type: user
paper_authors.user: JamesHujy
paper_authors._id: 663058fb33ccd9e68c4b47da
paper_authors.name: Shengding Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:38:51.615Z
paper_authors._id: 6041bec6c15e823a685f2afc
paper_authors.avatarUrl: /avatars/7d09fbe771d3c3312ac7eb6ef83ce053.svg
paper_authors.fullname: Shengding Hu
paper_authors.type: user
paper_authors.user: ShengdingHu
paper_authors._id: 663058fb33ccd9e68c4b47db
paper_authors.name: An Liu
paper_authors._id: 663058fb33ccd9e68c4b47dc
paper_authors.name: Yuge Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T07:39:16.974Z
paper_authors._id: 637f3220e0d5fd3199579458
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f3220e0d5fd3199579458/uaPvnmGqRcScz9RgoUcVY.png
paper_authors.fullname: Yuge Tu
paper_authors.type: user
paper_authors.user: tuyuge
paper_authors._id: 663058fb33ccd9e68c4b47dd
paper_authors.name: Pengkai Li
paper_authors._id: 663058fb33ccd9e68c4b47de
paper_authors.name: Lei Shi
paper_authors._id: 663058fb33ccd9e68c4b47df
paper_authors.name: Zhiyuan Liu
paper_authors._id: 663058fb33ccd9e68c4b47e0
paper_authors.name: Maosong Sun
paper_title: LEGENT: Open Platform for Embodied Agents
paper_summary: Despite advancements in Large Language Models (LLMs) and Large Multimodal
  Models (LMMs), their integration into language-grounded, human-like embodied
  agents remains incomplete, hindering complex real-life task performance in
  physical environments. Existing integrations often feature limited open
  sourcing, challenging collective progress in this field. We introduce LEGENT,
  an open, scalable platform for developing embodied agents using LLMs and LMMs.
  LEGENT offers a dual approach: a rich, interactive 3D environment with
  communicable and actionable agents, paired with a user-friendly interface, and
  a sophisticated data generation pipeline utilizing advanced algorithms to
  exploit supervision from simulated worlds at scale. In our experiments, an
  embryonic vision-language-action model trained on LEGENT-generated data
  surpasses GPT-4V in embodied tasks, showcasing promising generalization
  capabilities.
paper_ai_keywords: Large Language Models (LLMs) Large Multimodal Models (LMMs) embodied agents 3D environment communicable and actionable agents sophisticated data generation pipeline advanced algorithms simulation worlds vision-language-action model GPT-4V