title: EgoLife: Towards Egocentric Life Assistant
summary: We introduce EgoLife, a project to develop an egocentric life assistant that
  accompanies and enhances personal efficiency through AI-powered wearable
  glasses. To lay the foundation for this assistant, we conducted a comprehensive
  data collection study where six participants lived together for one week,
  continuously recording their daily activities - including discussions,
  shopping, cooking, socializing, and entertainment - using AI glasses for
  multimodal egocentric video capture, along with synchronized third-person-view
  video references. This effort resulted in the EgoLife Dataset, a comprehensive
  300-hour egocentric, interpersonal, multiview, and multimodal daily life
  dataset with intensive annotation. Leveraging this dataset, we introduce
  EgoLifeQA, a suite of long-context, life-oriented question-answering tasks
  designed to provide meaningful assistance in daily life by addressing practical
  questions such as recalling past relevant events, monitoring health habits, and
  offering personalized recommendations. To address the key technical challenges
  of (1) developing robust visual-audio models for egocentric data, (2) enabling
  identity recognition, and (3) facilitating long-context question answering over
  extensive temporal information, we introduce EgoButler, an integrated system
  comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on
  egocentric datasets, achieving state-of-the-art performance on egocentric video
  understanding. EgoRAG is a retrieval-based component that supports answering
  ultra-long-context questions. Our experimental studies verify their working
  mechanisms and reveal critical factors and bottlenecks, guiding future
  improvements. By releasing our datasets, models, and benchmarks, we aim to
  stimulate further research in egocentric AI assistants.
paper_authors._id: 67ca874c3ac187dbbed924d6
paper_authors.name: Jingkang Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T09:09:32.949Z
paper_authors._id: 62b5777f593a2c49da69dc02
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658152070753-62b5777f593a2c49da69dc02.jpeg
paper_authors.fullname: Jingkang Yang
paper_authors.type: user
paper_authors.user: Jingkang
paper_authors._id: 67ca874c3ac187dbbed924d7
paper_authors.name: Shuai Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T13:38:04.320Z
paper_authors._id: 64f7f5b54101c731ca84ae05
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f7f5b54101c731ca84ae05/13DwdxOo3tWbxKDLd44B9.jpeg
paper_authors.fullname: Shuai Liu
paper_authors.type: user
paper_authors.user: Choiszt
paper_authors._id: 67ca874c3ac187dbbed924d8
paper_authors.name: Hongming Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-17T16:33:53.816Z
paper_authors._id: 6638a424c8950692a4abc4e0
paper_authors.avatarUrl: /avatars/a135510b7b32596687f576ec5f23473f.svg
paper_authors.fullname: Hongming Guo
paper_authors.type: user
paper_authors.user: hongming03
paper_authors._id: 67ca874c3ac187dbbed924d9
paper_authors.name: Yuhao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:15:21.671Z
paper_authors._id: 652965773a416e1f2173443b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
paper_authors.fullname: Yuhao Dong
paper_authors.type: user
paper_authors.user: THUdyh
paper_authors._id: 67ca874c3ac187dbbed924da
paper_authors.name: Xiamengwei Zhang
paper_authors._id: 67ca874c3ac187dbbed924db
paper_authors.name: Sicheng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:15:37.377Z
paper_authors._id: 63f87c42b0ae1748524a9cfb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f87c42b0ae1748524a9cfb/I5ukv6iWoJVToWmcERmvx.jpeg
paper_authors.fullname: Sicheng Zhang
paper_authors.type: user
paper_authors.user: fesvhtr
paper_authors._id: 67ca874c3ac187dbbed924dc
paper_authors.name: Pengyun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:15:44.290Z
paper_authors._id: 62f113d3b58090c873d66481
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659966415211-noauth.jpeg
paper_authors.fullname: Pengyun Wang
paper_authors.type: user
paper_authors.user: Alarak
paper_authors._id: 67ca874c3ac187dbbed924dd
paper_authors.name: Zitang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:15:56.178Z
paper_authors._id: 668eb3a1a2f3f9d5edf029eb
paper_authors.avatarUrl: /avatars/383636e449f5e48c790f428818dd6863.svg
paper_authors.fullname: zhou zitang
paper_authors.type: user
paper_authors.user: Zzitang
paper_authors._id: 67ca874c3ac187dbbed924de
paper_authors.name: Binzhu Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:16:02.570Z
paper_authors._id: 63f886a99f87cc3e645c99a8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f886a99f87cc3e645c99a8/qwj16BrFaDjN0DPFsJ-6v.jpeg
paper_authors.fullname: Binzhu Xie
paper_authors.type: user
paper_authors.user: Nicous
paper_authors._id: 67ca874c3ac187dbbed924df
paper_authors.name: Ziyue Wang
paper_authors._id: 67ca874c3ac187dbbed924e0
paper_authors.name: Bei Ouyang
paper_authors._id: 67ca874c3ac187dbbed924e1
paper_authors.name: Zhengyu Lin
paper_authors._id: 67ca874c3ac187dbbed924e2
paper_authors.name: Marco Cominelli
paper_authors._id: 67ca874c3ac187dbbed924e3
paper_authors.name: Zhongang Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:16:33.666Z
paper_authors._id: 652d06833b5997ed71ce5c46
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/xZTXEcnEogEmBm_ledJQr.jpeg
paper_authors.fullname: Zhongang Cai
paper_authors.type: user
paper_authors.user: caizhongang
paper_authors._id: 67ca874c3ac187dbbed924e4
paper_authors.name: Yuanhan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:16:45.989Z
paper_authors._id: 62a993d80472c0b7f94027df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
paper_authors.fullname: Zhang Yuanhan
paper_authors.type: user
paper_authors.user: ZhangYuanhan
paper_authors._id: 67ca874c3ac187dbbed924e5
paper_authors.name: Peiyuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:16:57.532Z
paper_authors._id: 63565cc56d7fcf1bedb7d347
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
paper_authors.fullname: Zhang Peiyuan
paper_authors.type: user
paper_authors.user: PY007
paper_authors._id: 67ca874c3ac187dbbed924e6
paper_authors.name: Fangzhou Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:17:05.845Z
paper_authors._id: 67443675924e80c3c8807b40
paper_authors.avatarUrl: /avatars/fb45422391e51d2ad641f09c8535653c.svg
paper_authors.fullname: fangzhou HONG
paper_authors.type: user
paper_authors.user: h12345678
paper_authors._id: 67ca874c3ac187dbbed924e7
paper_authors.name: Joerg Widmer
paper_authors._id: 67ca874c3ac187dbbed924e8
paper_authors.name: Francesco Gringoli
paper_authors._id: 67ca874c3ac187dbbed924e9
paper_authors.name: Lei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T08:08:29.931Z
paper_authors._id: 6626a471430a124253f197c8
paper_authors.avatarUrl: /avatars/f5747fdbe495d1296fed9d16d8c95857.svg
paper_authors.fullname: yl-1993
paper_authors.type: user
paper_authors.user: yl-1993
paper_authors._id: 67ca874c3ac187dbbed924ea
paper_authors.name: Bo Li
paper_authors._id: 67ca874c3ac187dbbed924eb
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:15:05.677Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: EgoLife: Towards Egocentric Life Assistant
paper_summary: We introduce EgoLife, a project to develop an egocentric life assistant that
  accompanies and enhances personal efficiency through AI-powered wearable
  glasses. To lay the foundation for this assistant, we conducted a comprehensive
  data collection study where six participants lived together for one week,
  continuously recording their daily activities - including discussions,
  shopping, cooking, socializing, and entertainment - using AI glasses for
  multimodal egocentric video capture, along with synchronized third-person-view
  video references. This effort resulted in the EgoLife Dataset, a comprehensive
  300-hour egocentric, interpersonal, multiview, and multimodal daily life
  dataset with intensive annotation. Leveraging this dataset, we introduce
  EgoLifeQA, a suite of long-context, life-oriented question-answering tasks
  designed to provide meaningful assistance in daily life by addressing practical
  questions such as recalling past relevant events, monitoring health habits, and
  offering personalized recommendations. To address the key technical challenges
  of (1) developing robust visual-audio models for egocentric data, (2) enabling
  identity recognition, and (3) facilitating long-context question answering over
  extensive temporal information, we introduce EgoButler, an integrated system
  comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on
  egocentric datasets, achieving state-of-the-art performance on egocentric video
  understanding. EgoRAG is a retrieval-based component that supports answering
  ultra-long-context questions. Our experimental studies verify their working
  mechanisms and reveal critical factors and bottlenecks, guiding future
  improvements. By releasing our datasets, models, and benchmarks, we aim to
  stimulate further research in egocentric AI assistants.
paper_ai_keywords: egocentric life assistant AI-powered wearable glasses egocentric video capture third-person-view video EgoLife Dataset life-oriented question-answering tasks long-context question answering EgoButler EgoGPT EgoRAG egocentric video understanding retrieval-based component ultra-long-context questions egocentric datasets omni-modal model identity recognition