title: EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language
    Models for Vision-Driven Embodied Agents
summary: Leveraging Multi-modal Large Language Models (MLLMs) to create embodied
  agents offers a promising avenue for tackling real-world tasks. While
  language-centric embodied agents have garnered substantial attention,
  MLLM-based embodied agents remain underexplored due to the lack of
  comprehensive evaluation frameworks. To bridge this gap, we introduce
  EmbodiedBench, an extensive benchmark designed to evaluate vision-driven
  embodied agents. EmbodiedBench features: (1) a diverse set of 1,128 testing
  tasks across four environments, ranging from high-level semantic tasks (e.g.,
  household) to low-level tasks involving atomic actions (e.g., navigation and
  manipulation); and (2) six meticulously curated subsets evaluating essential
  agent capabilities like commonsense reasoning, complex instruction
  understanding, spatial awareness, visual perception, and long-term planning.
  Through extensive experiments, we evaluated 13 leading proprietary and
  open-source MLLMs within EmbodiedBench. Our findings reveal that: MLLMs excel
  at high-level tasks but struggle with low-level manipulation, with the best
  model, GPT-4o, scoring only 28.9% on average. EmbodiedBench provides a
  multifaceted standardized evaluation platform that not only highlights existing
  challenges but also offers valuable insights to advance MLLM-based embodied
  agents. Our code is available at https://embodiedbench.github.io.
paper_authors._id: 67aec4285b9801b819449b84
paper_authors.name: Rui Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:34:05.913Z
paper_authors._id: 64d45451c34a346181b130dd
paper_authors.avatarUrl: /avatars/9bb8205b889337df5d321539c9b5d69d.svg
paper_authors.fullname: Rui Yang
paper_authors.type: user
paper_authors.user: Ray2333
paper_authors._id: 67aec4285b9801b819449b85
paper_authors.name: Hanyang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T08:01:08.365Z
paper_authors._id: 6700b1f93381f2db06857fb5
paper_authors.avatarUrl: /avatars/c8b9ec7c00773c5a4055ba50de0c6b2f.svg
paper_authors.fullname: Hanyang Chen
paper_authors.type: user
paper_authors.user: Hanyang81
paper_authors._id: 67aec4285b9801b819449b86
paper_authors.name: Junyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:34:03.521Z
paper_authors._id: 6719bfd07c6e6c83a388aeae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6719bfd07c6e6c83a388aeae/jHxryk04dzHo23TX5F5sz.png
paper_authors.fullname: Junyu Zhang
paper_authors.type: user
paper_authors.user: jyzhang1208
paper_authors._id: 67aec4285b9801b819449b87
paper_authors.name: Mark Zhao
paper_authors._id: 67aec4285b9801b819449b88
paper_authors.name: Cheng Qian
paper_authors._id: 67aec4285b9801b819449b89
paper_authors.name: Kangrui Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:47:36.061Z
paper_authors._id: 66f2b8602ef2817ec3cb65f6
paper_authors.avatarUrl: /avatars/c8c5b2706644fb45a75f13af99fa7ae9.svg
paper_authors.fullname: Kangrui Wang
paper_authors.type: user
paper_authors.user: JamesK2W
paper_authors._id: 67aec4285b9801b819449b8a
paper_authors.name: Qineng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:46:06.895Z
paper_authors._id: 640c3b28c5fa12d61a50cd92
paper_authors.avatarUrl: /avatars/81556de3214c848b3c3e118f50fd2968.svg
paper_authors.fullname: Qineng Wang
paper_authors.type: user
paper_authors.user: Inevitablevalor
paper_authors._id: 67aec4285b9801b819449b8b
paper_authors.name: Teja Venkat Koripella
paper_authors._id: 67aec4285b9801b819449b8c
paper_authors.name: Marziyeh Movahedi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:46:13.104Z
paper_authors._id: 64e10c263209bf4194912319
paper_authors.avatarUrl: /avatars/02f1a9e2ce333ff521d901cf83fcdff3.svg
paper_authors.fullname: Marziyeh Movahedi
paper_authors.type: user
paper_authors.user: Marzimv
paper_authors._id: 67aec4285b9801b819449b8d
paper_authors.name: Manling Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:46:50.911Z
paper_authors._id: 6746a140f2ca2162e3bcfe2b
paper_authors.avatarUrl: /avatars/d9d8cfb5f112e6ed7f6152fc230135d3.svg
paper_authors.fullname: Manling Li
paper_authors.type: user
paper_authors.user: ManlingLi
paper_authors._id: 67aec4285b9801b819449b8e
paper_authors.name: Heng Ji
paper_authors._id: 67aec4285b9801b819449b8f
paper_authors.name: Huan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:47:14.741Z
paper_authors._id: 6719d581a6cad13741b8bc7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6719d581a6cad13741b8bc7f/w4EttqfXRgWZJc6HpYOS9.jpeg
paper_authors.fullname: Huan Zhang
paper_authors.type: user
paper_authors.user: huanzhang12
paper_authors._id: 67aec4285b9801b819449b90
paper_authors.name: Tong Zhang
paper_title: EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language
    Models for Vision-Driven Embodied Agents
paper_summary: Leveraging Multi-modal Large Language Models (MLLMs) to create embodied
  agents offers a promising avenue for tackling real-world tasks. While
  language-centric embodied agents have garnered substantial attention,
  MLLM-based embodied agents remain underexplored due to the lack of
  comprehensive evaluation frameworks. To bridge this gap, we introduce
  EmbodiedBench, an extensive benchmark designed to evaluate vision-driven
  embodied agents. EmbodiedBench features: (1) a diverse set of 1,128 testing
  tasks across four environments, ranging from high-level semantic tasks (e.g.,
  household) to low-level tasks involving atomic actions (e.g., navigation and
  manipulation); and (2) six meticulously curated subsets evaluating essential
  agent capabilities like commonsense reasoning, complex instruction
  understanding, spatial awareness, visual perception, and long-term planning.
  Through extensive experiments, we evaluated 13 leading proprietary and
  open-source MLLMs within EmbodiedBench. Our findings reveal that: MLLMs excel
  at high-level tasks but struggle with low-level manipulation, with the best
  model, GPT-4o, scoring only 28.9% on average. EmbodiedBench provides a
  multifaceted standardized evaluation platform that not only highlights existing
  challenges but also offers valuable insights to advance MLLM-based embodied
  agents. Our code is available at https://embodiedbench.github.io.
paper_ai_keywords: Multi-modal Large Language Models (MLLMs) embodied agents EmbodiedBench vision-driven commonsense reasoning complex instruction understanding spatial awareness visual perception long-term planning