title: EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
summary: Multimodal Large Language Models (MLLMs) have shown significant advancements,
  providing a promising future for embodied agents. Existing benchmarks for
  evaluating MLLMs primarily utilize static images or videos, limiting
  assessments to non-interactive scenarios. Meanwhile, existing embodied AI
  benchmarks are task-specific and not diverse enough, which do not adequately
  evaluate the embodied capabilities of MLLMs. To address this, we propose
  EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs
  with embodied tasks. EmbodiedEval features 328 distinct tasks within 125 varied
  3D scenes, each of which is rigorously selected and annotated. It covers a
  broad spectrum of existing embodied AI tasks with significantly enhanced
  diversity, all within a unified simulation and evaluation framework tailored
  for MLLMs. The tasks are organized into five categories: navigation, object
  interaction, social interaction, attribute question answering, and spatial
  question answering to assess different capabilities of the agents. We evaluated
  the state-of-the-art MLLMs on EmbodiedEval and found that they have a
  significant shortfall compared to human level on embodied tasks. Our analysis
  demonstrates the limitations of existing MLLMs in embodied capabilities,
  providing insights for their future development. We open-source all evaluation
  data and simulation framework at https://github.com/thunlp/EmbodiedEval.
paper_authors._id: 67943e877aa0faad86e4d750
paper_authors.name: Zhili Cheng
paper_authors._id: 67943e877aa0faad86e4d751
paper_authors.name: Yuge Tu
paper_authors._id: 67943e877aa0faad86e4d752
paper_authors.name: Ran Li
paper_authors._id: 67943e877aa0faad86e4d753
paper_authors.name: Shiqi Dai
paper_authors._id: 67943e877aa0faad86e4d754
paper_authors.name: Jinyi Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-26T11:38:36.572Z
paper_authors._id: 637a06580a77f602dc4ac922
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637a06580a77f602dc4ac922/JR2VHWhG1tj3TKKNzD8GD.jpeg
paper_authors.fullname: Jinyi Hu
paper_authors.type: user
paper_authors.user: JamesHujy
paper_authors._id: 67943e877aa0faad86e4d755
paper_authors.name: Shengding Hu
paper_authors._id: 67943e877aa0faad86e4d756
paper_authors.name: Jiahao Li
paper_authors._id: 67943e877aa0faad86e4d757
paper_authors.name: Yang Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:35:10.394Z
paper_authors._id: 673c7319d11b1c2e246ead9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
paper_authors.fullname: Yang Shi
paper_authors.type: user
paper_authors.user: DogNeverSleep
paper_authors._id: 67943e877aa0faad86e4d758
paper_authors.name: Tianyu Yu
paper_authors._id: 67943e877aa0faad86e4d759
paper_authors.name: Weize Chen
paper_authors._id: 67943e877aa0faad86e4d75a
paper_authors.name: Lei Shi
paper_authors._id: 67943e877aa0faad86e4d75b
paper_authors.name: Maosong Sun
paper_title: EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
paper_summary: Multimodal Large Language Models (MLLMs) have shown significant advancements,
  providing a promising future for embodied agents. Existing benchmarks for
  evaluating MLLMs primarily utilize static images or videos, limiting
  assessments to non-interactive scenarios. Meanwhile, existing embodied AI
  benchmarks are task-specific and not diverse enough, which do not adequately
  evaluate the embodied capabilities of MLLMs. To address this, we propose
  EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs
  with embodied tasks. EmbodiedEval features 328 distinct tasks within 125 varied
  3D scenes, each of which is rigorously selected and annotated. It covers a
  broad spectrum of existing embodied AI tasks with significantly enhanced
  diversity, all within a unified simulation and evaluation framework tailored
  for MLLMs. The tasks are organized into five categories: navigation, object
  interaction, social interaction, attribute question answering, and spatial
  question answering to assess different capabilities of the agents. We evaluated
  the state-of-the-art MLLMs on EmbodiedEval and found that they have a
  significant shortfall compared to human level on embodied tasks. Our analysis
  demonstrates the limitations of existing MLLMs in embodied capabilities,
  providing insights for their future development. We open-source all evaluation
  data and simulation framework at https://github.com/thunlp/EmbodiedEval.
paper_ai_keywords: multimodal large language models MLLMs embodied agents benchmarks static images videos non-interactive scenarios embodied AI benchmarks task-specific comprehensive interactive evaluation benchmark 328 distinct tasks 125 varied 3D scenes unified simulation evaluation framework navigation object interaction social interaction attribute question answering spatial question answering embodied tasks human level embodied capabilities open-source