title: VidEgoThink: Assessing Egocentric Video Understanding Capabilities for
    Embodied AI
summary: Recent advancements in Multi-modal Large Language Models (MLLMs) have opened
  new avenues for applications in Embodied AI. Building on previous work,
  EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating
  egocentric video understanding capabilities. To bridge the gap between MLLMs
  and low-level control in Embodied AI, we design four key interrelated tasks:
  video question-answering, hierarchy planning, visual grounding and reward
  modeling. To minimize manual annotation costs, we develop an automatic data
  generation pipeline based on the Ego4D dataset, leveraging the prior knowledge
  and multimodal capabilities of GPT-4o. Three human annotators then filter the
  generated data to ensure diversity and quality, resulting in the VidEgoThink
  benchmark. We conduct extensive experiments with three types of models:
  API-based MLLMs, open-source image-based MLLMs, and open-source video-based
  MLLMs. Experimental results indicate that all MLLMs, including GPT-4o, perform
  poorly across all tasks related to egocentric video understanding. These
  findings suggest that foundation models still require significant advancements
  to be effectively applied to first-person scenarios in Embodied AI. In
  conclusion, VidEgoThink reflects a research trend towards employing MLLMs for
  egocentric vision, akin to human capabilities, enabling active observation and
  interaction in the complex real-world environments.
paper_authors._id: 6710b440eac1980726739204
paper_authors.name: Sijie Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:24:13.175Z
paper_authors._id: 639003391c5a623727252735
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639003391c5a623727252735/fp_wZEhGzxGlyr64XF-x9.jpeg
paper_authors.fullname: Sijie Cheng
paper_authors.type: user
paper_authors.user: SijieCheng
paper_authors._id: 6710b440eac1980726739205
paper_authors.name: Kechen Fang
paper_authors._id: 6710b440eac1980726739206
paper_authors.name: Yangyang Yu
paper_authors._id: 6710b440eac1980726739207
paper_authors.name: Sicheng Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:14:23.086Z
paper_authors._id: 663c83439b8151660f29573e
paper_authors.avatarUrl: /avatars/89b52fa42e198b79e4d5bd069c798d31.svg
paper_authors.fullname: Sicheng Zhou
paper_authors.type: user
paper_authors.user: eternity304
paper_authors._id: 6710b440eac1980726739208
paper_authors.name: Bohao Li
paper_authors._id: 6710b440eac1980726739209
paper_authors.name: Ye Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:24:11.041Z
paper_authors._id: 6414061f996b2e426f23df48
paper_authors.avatarUrl: /avatars/db0af49c5f017f7c455a076b8b2212b6.svg
paper_authors.fullname: Ye Tian
paper_authors.type: user
paper_authors.user: yetian
paper_authors._id: 6710b440eac198072673920a
paper_authors.name: Tingguang Li
paper_authors._id: 6710b440eac198072673920b
paper_authors.name: Lei Han
paper_authors._id: 6710b440eac198072673920c
paper_authors.name: Yang Liu
paper_title: VidEgoThink: Assessing Egocentric Video Understanding Capabilities for
    Embodied AI
paper_summary: Recent advancements in Multi-modal Large Language Models (MLLMs) have opened
  new avenues for applications in Embodied AI. Building on previous work,
  EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating
  egocentric video understanding capabilities. To bridge the gap between MLLMs
  and low-level control in Embodied AI, we design four key interrelated tasks:
  video question-answering, hierarchy planning, visual grounding and reward
  modeling. To minimize manual annotation costs, we develop an automatic data
  generation pipeline based on the Ego4D dataset, leveraging the prior knowledge
  and multimodal capabilities of GPT-4o. Three human annotators then filter the
  generated data to ensure diversity and quality, resulting in the VidEgoThink
  benchmark. We conduct extensive experiments with three types of models:
  API-based MLLMs, open-source image-based MLLMs, and open-source video-based
  MLLMs. Experimental results indicate that all MLLMs, including GPT-4o, perform
  poorly across all tasks related to egocentric video understanding. These
  findings suggest that foundation models still require significant advancements
  to be effectively applied to first-person scenarios in Embodied AI. In
  conclusion, VidEgoThink reflects a research trend towards employing MLLMs for
  egocentric vision, akin to human capabilities, enabling active observation and
  interaction in the complex real-world environments.
paper_ai_keywords: Multi-modal Large Language Models (MLLMs) Embodied AI VidEgoThink egocentric video understanding video question-answering hierarchy planning visual grounding reward modeling automatic data generation pipeline Ego4D dataset GPT-4o API-based MLLMs open-source image-based MLLMs open-source video-based MLLMs foundation models first-person scenarios