EgoLife: Towards Egocentric Life Assistant
We introduce EgoLife, a project to develop an egocentric life assistant that
accompanies and enhances personal efficiency through AI-powered wearable
glasses. To lay the foundation for this assistant, we conducted a comprehensive
data collection study where six participants lived together for one week,
continuously recording their daily activities - including discussions,
shopping, cooking, socializing, and entertainment - using AI glasses for
multimodal egocentric video capture, along with synchronized third-person-view
video references. This effort resulted in the EgoLife Dataset, a comprehensive
300-hour egocentric, interpersonal, multiview, and multimodal daily life
dataset with intensive annotation. Leveraging this dataset, we introduce
EgoLifeQA, a suite of long-context, life-oriented question-answering tasks
designed to provide meaningful assistance in daily life by addressing practical
questions such as recalling past relevant events, monitoring health habits, and
offering personalized recommendations. To address the key technical challenges
of (1) developing robust visual-audio models for egocentric data, (2) enabling
identity recognition, and (3) facilitating long-context question answering over
extensive temporal information, we introduce EgoButler, an integrated system
comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on
egocentric datasets, achieving state-of-the-art performance on egocentric video
understanding. EgoRAG is a retrieval-based component that supports answering
ultra-long-context questions. Our experimental studies verify their working
mechanisms and reveal critical factors and bottlenecks, guiding future
improvements. By releasing our datasets, models, and benchmarks, we aim to
stimulate further research in egocentric AI assistants.
67ca874c3ac187dbbed924d6
Jingkang Yang
claimed_verified
2025-03-07T09:09:32.949Z
62b5777f593a2c49da69dc02
https://cdn-avatars.huggingface.co/v1/production/uploads/1658152070753-62b5777f593a2c49da69dc02.jpeg
Jingkang Yang
user
Jingkang
67ca874c3ac187dbbed924d7
Shuai Liu
claimed_verified
2025-03-07T13:38:04.320Z
64f7f5b54101c731ca84ae05
https://cdn-avatars.huggingface.co/v1/production/uploads/64f7f5b54101c731ca84ae05/13DwdxOo3tWbxKDLd44B9.jpeg
Shuai Liu
user
Choiszt
67ca874c3ac187dbbed924d8
Hongming Guo
claimed_verified
2025-03-17T16:33:53.816Z
6638a424c8950692a4abc4e0
/avatars/a135510b7b32596687f576ec5f23473f.svg
Hongming Guo
user
hongming03
67ca874c3ac187dbbed924d9
Yuhao Dong
admin_assigned
2025-03-07T09:15:21.671Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
user
THUdyh
67ca874c3ac187dbbed924da
Xiamengwei Zhang
67ca874c3ac187dbbed924db
Sicheng Zhang
admin_assigned
2025-03-07T09:15:37.377Z
63f87c42b0ae1748524a9cfb
https://cdn-avatars.huggingface.co/v1/production/uploads/63f87c42b0ae1748524a9cfb/I5ukv6iWoJVToWmcERmvx.jpeg
Sicheng Zhang
user
fesvhtr
67ca874c3ac187dbbed924dc
Pengyun Wang
admin_assigned
2025-03-07T09:15:44.290Z
62f113d3b58090c873d66481
https://cdn-avatars.huggingface.co/v1/production/uploads/1659966415211-noauth.jpeg
Pengyun Wang
user
Alarak
67ca874c3ac187dbbed924dd
Zitang Zhou
admin_assigned
2025-03-07T09:15:56.178Z
668eb3a1a2f3f9d5edf029eb
/avatars/383636e449f5e48c790f428818dd6863.svg
zhou zitang
user
Zzitang
67ca874c3ac187dbbed924de
Binzhu Xie
admin_assigned
2025-03-07T09:16:02.570Z
63f886a99f87cc3e645c99a8
https://cdn-avatars.huggingface.co/v1/production/uploads/63f886a99f87cc3e645c99a8/qwj16BrFaDjN0DPFsJ-6v.jpeg
Binzhu Xie
user
Nicous
67ca874c3ac187dbbed924df
Ziyue Wang
67ca874c3ac187dbbed924e0
Bei Ouyang
67ca874c3ac187dbbed924e1
Zhengyu Lin
67ca874c3ac187dbbed924e2
Marco Cominelli
67ca874c3ac187dbbed924e3
Zhongang Cai
admin_assigned
2025-03-07T09:16:33.666Z
652d06833b5997ed71ce5c46
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/xZTXEcnEogEmBm_ledJQr.jpeg
Zhongang Cai
user
caizhongang
67ca874c3ac187dbbed924e4
Yuanhan Zhang
admin_assigned
2025-03-07T09:16:45.989Z
62a993d80472c0b7f94027df
https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
Zhang Yuanhan
user
ZhangYuanhan
67ca874c3ac187dbbed924e5
Peiyuan Zhang
admin_assigned
2025-03-07T09:16:57.532Z
63565cc56d7fcf1bedb7d347
https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
Zhang Peiyuan
user
PY007
67ca874c3ac187dbbed924e6
Fangzhou Hong
admin_assigned
2025-03-07T09:17:05.845Z
67443675924e80c3c8807b40
/avatars/fb45422391e51d2ad641f09c8535653c.svg
fangzhou HONG
user
h12345678
67ca874c3ac187dbbed924e7
Joerg Widmer
67ca874c3ac187dbbed924e8
Francesco Gringoli
67ca874c3ac187dbbed924e9
Lei Yang
admin_assigned
2025-03-10T08:08:29.931Z
6626a471430a124253f197c8
/avatars/f5747fdbe495d1296fed9d16d8c95857.svg
yl-1993
user
yl-1993
67ca874c3ac187dbbed924ea
Bo Li
claimed_verified
2025-06-27T20:31:21.630Z
62d3f7d84b0933c48f3cdd9c
https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
Bo Li
user
luodian
67ca874c3ac187dbbed924eb
Ziwei Liu
admin_assigned
2025-03-07T09:15:05.677Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
EgoLife: Towards Egocentric Life Assistant
We introduce EgoLife, a project to develop an egocentric life assistant that
accompanies and enhances personal efficiency through AI-powered wearable
glasses. To lay the foundation for this assistant, we conducted a comprehensive
data collection study where six participants lived together for one week,
continuously recording their daily activities - including discussions,
shopping, cooking, socializing, and entertainment - using AI glasses for
multimodal egocentric video capture, along with synchronized third-person-view
video references. This effort resulted in the EgoLife Dataset, a comprehensive
300-hour egocentric, interpersonal, multiview, and multimodal daily life
dataset with intensive annotation. Leveraging this dataset, we introduce
EgoLifeQA, a suite of long-context, life-oriented question-answering tasks
designed to provide meaningful assistance in daily life by addressing practical
questions such as recalling past relevant events, monitoring health habits, and
offering personalized recommendations. To address the key technical challenges
of (1) developing robust visual-audio models for egocentric data, (2) enabling
identity recognition, and (3) facilitating long-context question answering over
extensive temporal information, we introduce EgoButler, an integrated system
comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on
egocentric datasets, achieving state-of-the-art performance on egocentric video
understanding. EgoRAG is a retrieval-based component that supports answering
ultra-long-context questions. Our experimental studies verify their working
mechanisms and reveal critical factors and bottlenecks, guiding future
improvements. By releasing our datasets, models, and benchmarks, we aim to
stimulate further research in egocentric AI assistants.
EgoLife introduces EgoButler, combining EgoGPT and EgoRAG, to provide AI-powered assistance through wearable glasses using a comprehensive egocentric dataset.
AI-powered wearable glasses egocentric video capture multimodal dataset EgoLife Dataset EgoLifeQA EgoButler EgoGPT EgoRAG egocentric video understanding ultra-long-context questions identity recognition omni-modal model retrieval-based component