title: EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
summary: Multimodal Large Language Models (MLLMs) have shown significant advancements,
  providing a promising future for embodied agents. Existing benchmarks for
  evaluating MLLMs primarily utilize static images or videos, limiting
  assessments to non-interactive scenarios. Meanwhile, existing embodied AI
  benchmarks are task-specific and not diverse enough, which do not adequately
  evaluate the embodied capabilities of MLLMs. To address this, we propose
  EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs
  with embodied tasks. EmbodiedEval features 328 distinct tasks within 125 varied
  3D scenes, each of which is rigorously selected and annotated. It covers a
  broad spectrum of existing embodied AI tasks with significantly enhanced
  diversity, all within a unified simulation and evaluation framework tailored
  for MLLMs. The tasks are organized into five categories: navigation, object
  interaction, social interaction, attribute question answering, and spatial
  question answering to assess different capabilities of the agents. We evaluated
  the state-of-the-art MLLMs on EmbodiedEval and found that they have a
  significant shortfall compared to human level on embodied tasks. Our analysis
  demonstrates the limitations of existing MLLMs in embodied capabilities,
  providing insights for their future development. We open-source all evaluation
  data and simulation framework at https://github.com/thunlp/EmbodiedEval.
paper_authors._id: 67943e877aa0faad86e4d750
paper_authors.name: Zhili Cheng
paper_authors._id: 67943e877aa0faad86e4d751
paper_authors.name: Yuge Tu
paper_authors._id: 67943e877aa0faad86e4d752
paper_authors.name: Ran Li
paper_authors._id: 67943e877aa0faad86e4d753
paper_authors.name: Shiqi Dai
paper_authors._id: 67943e877aa0faad86e4d754
paper_authors.name: Jinyi Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-26T11:38:36.572Z
paper_authors._id: 637a06580a77f602dc4ac922
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637a06580a77f602dc4ac922/JR2VHWhG1tj3TKKNzD8GD.jpeg
paper_authors.fullname: Jinyi Hu
paper_authors.type: user
paper_authors.user: JamesHujy
paper_authors._id: 67943e877aa0faad86e4d755
paper_authors.name: Shengding Hu
paper_authors._id: 67943e877aa0faad86e4d756
paper_authors.name: Jiahao Li
paper_authors._id: 67943e877aa0faad86e4d757
paper_authors.name: Yang Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:35:10.394Z
paper_authors._id: 673c7319d11b1c2e246ead9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
paper_authors.fullname: Yang Shi
paper_authors.type: user
paper_authors.user: DogNeverSleep
paper_authors._id: 67943e877aa0faad86e4d758
paper_authors.name: Tianyu Yu
paper_authors._id: 67943e877aa0faad86e4d759
paper_authors.name: Weize Chen
paper_authors._id: 67943e877aa0faad86e4d75a
paper_authors.name: Lei Shi
paper_authors._id: 67943e877aa0faad86e4d75b
paper_authors.name: Maosong Sun
paper_title: EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
paper_summary: Multimodal Large Language Models (MLLMs) have shown significant advancements,
  providing a promising future for embodied agents. Existing benchmarks for
  evaluating MLLMs primarily utilize static images or videos, limiting
  assessments to non-interactive scenarios. Meanwhile, existing embodied AI
  benchmarks are task-specific and not diverse enough, which do not adequately
  evaluate the embodied capabilities of MLLMs. To address this, we propose
  EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs
  with embodied tasks. EmbodiedEval features 328 distinct tasks within 125 varied
  3D scenes, each of which is rigorously selected and annotated. It covers a
  broad spectrum of existing embodied AI tasks with significantly enhanced
  diversity, all within a unified simulation and evaluation framework tailored
  for MLLMs. The tasks are organized into five categories: navigation, object
  interaction, social interaction, attribute question answering, and spatial
  question answering to assess different capabilities of the agents. We evaluated
  the state-of-the-art MLLMs on EmbodiedEval and found that they have a
  significant shortfall compared to human level on embodied tasks. Our analysis
  demonstrates the limitations of existing MLLMs in embodied capabilities,
  providing insights for their future development. We open-source all evaluation
  data and simulation framework at https://github.com/thunlp/EmbodiedEval.
paper_ai_summary: EmbodiedEval is a comprehensive benchmark for assessing the embodied capabilities of multimodal large language models using diverse interactive tasks in 3D scenes.
paper_ai_keywords: multimodal large language models embodied AI static images videos non-interactive scenarios task-specific benchmarks embodiedEval 328 distinct tasks 125 varied 3D scenes navigation object interaction social interaction attribute question answering spatial question answering state-of-the-art MLLMs