title: MM-Ego: Towards Building Egocentric Multimodal LLMs
summary: This research aims to comprehensively explore building a multimodal
  foundation model for egocentric video understanding. To achieve this goal, we
  work on three fronts. First, as there is a lack of QA data for egocentric video
  understanding, we develop a data engine that efficiently generates 7M
  high-quality QA samples for egocentric videos ranging from 30 seconds to one
  hour long, based on human-annotated data. This is currently the largest
  egocentric QA dataset. Second, we contribute a challenging egocentric QA
  benchmark with 629 videos and 7,026 questions to evaluate the models' ability
  in recognizing and memorizing visual details across videos of varying lengths.
  We introduce a new de-biasing evaluation method to help mitigate the
  unavoidable language bias present in the models being evaluated. Third, we
  propose a specialized multimodal architecture featuring a novel "Memory Pointer
  Prompting" mechanism. This design includes a global glimpse step to gain an
  overarching understanding of the entire video and identify key visual
  information, followed by a fallback step that utilizes the key visual
  information to generate responses. This enables the model to more effectively
  comprehend extended video content. With the data, benchmark, and model, we
  successfully build MM-Ego, an egocentric multimodal LLM that shows powerful
  performance on egocentric video understanding.
paper_authors._id: 67074fe8784e1c990108328a
paper_authors.name: Hanrong Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:56:29.882Z
paper_authors._id: 62d3ae4d894e7fe42def988f
paper_authors.avatarUrl: /avatars/3aafc55d9783459f9a79546fc31dd68a.svg
paper_authors.fullname: Hanrong Ye
paper_authors.type: user
paper_authors.user: leoye
paper_authors._id: 67074fe8784e1c990108328b
paper_authors.name: Haotian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:00:37.565Z
paper_authors._id: 631516348d85ad332fa47b2c
paper_authors.avatarUrl: /avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
paper_authors.fullname: Haotian Zhang
paper_authors.type: user
paper_authors.user: haotiz
paper_authors._id: 67074fe8784e1c990108328c
paper_authors.name: Erik Daxberger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:56:45.821Z
paper_authors._id: 66150e514476535462a7166c
paper_authors.avatarUrl: /avatars/75a0b6546dde9853a06c9536f067005d.svg
paper_authors.fullname: Erik Daxberger
paper_authors.type: user
paper_authors.user: edaxberger
paper_authors._id: 67074fe8784e1c990108328d
paper_authors.name: Lin Chen
paper_authors._id: 67074fe8784e1c990108328e
paper_authors.name: Zongyu Lin
paper_authors._id: 67074fe8784e1c990108328f
paper_authors.name: Yanghao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:58:00.320Z
paper_authors._id: 65dad3870af7e21ba473439f
paper_authors.avatarUrl: /avatars/da542e7d68ae937bbdb791f17096bb1c.svg
paper_authors.fullname: Yanghao Li
paper_authors.type: user
paper_authors.user: FrozzZen
paper_authors._id: 67074fe8784e1c9901083290
paper_authors.name: Bowen Zhang
paper_authors._id: 67074fe8784e1c9901083291
paper_authors.name: Haoxuan You
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:57:36.252Z
paper_authors._id: 65284ba3e3b3844f5e487a0b
paper_authors.avatarUrl: /avatars/fce12a233bddb611d441f25a2ab17725.svg
paper_authors.fullname: Haoxuan You
paper_authors.type: user
paper_authors.user: HaoxuanYou
paper_authors._id: 67074fe8784e1c9901083292
paper_authors.name: Dan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:57:29.789Z
paper_authors._id: 66feab48651e00e22f33222e
paper_authors.avatarUrl: /avatars/7344377e2c796c7ec85194bb2fc78521.svg
paper_authors.fullname: Dan Xu
paper_authors.type: user
paper_authors.user: danxuhk
paper_authors._id: 67074fe8784e1c9901083293
paper_authors.name: Zhe Gan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:57:03.855Z
paper_authors._id: 644022db21ec7c165d848b6e
paper_authors.avatarUrl: /avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
paper_authors.fullname: Zhe Gan
paper_authors.type: user
paper_authors.user: zhegan27
paper_authors._id: 67074fe8784e1c9901083294
paper_authors.name: Jiasen Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:56:57.823Z
paper_authors._id: 62b6b0397523238923221df9
paper_authors.avatarUrl: /avatars/77068771dd51df7519516cd502a88789.svg
paper_authors.fullname: Jiasenlu
paper_authors.type: user
paper_authors.user: Jiasenlu
paper_authors._id: 67074fe8784e1c9901083295
paper_authors.name: Yinfei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:56:52.265Z
paper_authors._id: 64b762568c632fbca942a405
paper_authors.avatarUrl: /avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
paper_authors.fullname: Yinfei Yang
paper_authors.type: user
paper_authors.user: yinfeiy
paper_title: MM-Ego: Towards Building Egocentric Multimodal LLMs
paper_summary: This research aims to comprehensively explore building a multimodal
  foundation model for egocentric video understanding. To achieve this goal, we
  work on three fronts. First, as there is a lack of QA data for egocentric video
  understanding, we develop a data engine that efficiently generates 7M
  high-quality QA samples for egocentric videos ranging from 30 seconds to one
  hour long, based on human-annotated data. This is currently the largest
  egocentric QA dataset. Second, we contribute a challenging egocentric QA
  benchmark with 629 videos and 7,026 questions to evaluate the models' ability
  in recognizing and memorizing visual details across videos of varying lengths.
  We introduce a new de-biasing evaluation method to help mitigate the
  unavoidable language bias present in the models being evaluated. Third, we
  propose a specialized multimodal architecture featuring a novel "Memory Pointer
  Prompting" mechanism. This design includes a global glimpse step to gain an
  overarching understanding of the entire video and identify key visual
  information, followed by a fallback step that utilizes the key visual
  information to generate responses. This enables the model to more effectively
  comprehend extended video content. With the data, benchmark, and model, we
  successfully build MM-Ego, an egocentric multimodal LLM that shows powerful
  performance on egocentric video understanding.
paper_ai_keywords: multimodal foundation model egocentric video understanding QA data data engine textual QA samples egocentric QA dataset semantic benchmark de-biasing evaluation method multimodal architecture Memory Pointer Prompting Mechanism global glimpse step fallback step extended video content MM-Ego multimodal LLM