Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning
We introduce Ego-R1, a novel framework for reasoning over ultra-long (i.e.,
in days and weeks) egocentric videos, which leverages a structured
Chain-of-Tool-Thought (CoTT) process, orchestrated by an Ego-R1 Agent trained
via reinforcement learning (RL). Inspired by human problem-solving strategies,
CoTT decomposes complex reasoning into modular steps, with the RL agent
invoking specific tools, one per step, to iteratively and collaboratively
answer sub-questions tackling such tasks as temporal retrieval and multi-modal
understanding. We design a two-stage training paradigm involving supervised
finetuning (SFT) of a pretrained language model using CoTT data and RL to
enable our agent to dynamically propose step-by-step tools for long-range
reasoning. To facilitate training, we construct a dataset called Ego-R1 Data,
which consists of Ego-CoTT-25K for SFT and Ego-QA-4.4K for RL. Furthermore, our
Ego-R1 agent is evaluated on a newly curated week-long video QA benchmark,
Ego-R1 Bench, which contains human-verified QA pairs from hybrid sources.
Extensive results demonstrate that the dynamic, tool-augmented chain-of-thought
reasoning by our Ego-R1 Agent can effectively tackle the unique challenges of
understanding ultra-long egocentric videos, significantly extending the time
coverage from few hours to a week.
6850e2a05e07650ecce89106
Shulin Tian
claimed_verified
2025-06-17T07:21:00.696Z
6658d01c6f1a71ba56d6c273
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tc4nZrMuZQLfgt5aVxtH4.jpeg
Tian Shulin
user
shulin16
6850e2a05e07650ecce89107
Ruiqi Wang
claimed_verified
2025-06-17T07:21:03.098Z
6303e551d14428368d194477
/avatars/b3c583e4525747b314379a7613e3b115.svg
Ruiqi Wang
user
ruiqiw
6850e2a05e07650ecce89108
Hongming Guo
6850e2a05e07650ecce89109
Penghao Wu
6850e2a05e07650ecce8910a
Yuhao Dong
claimed_verified
2025-06-23T08:29:23.359Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
user
THUdyh
6850e2a05e07650ecce8910b
Xiuying Wang
6850e2a05e07650ecce8910c
Jingkang Yang
6850e2a05e07650ecce8910d
Hao Zhang
6850e2a05e07650ecce8910e
Hongyuan Zhu
6850e2a05e07650ecce8910f
Ziwei Liu
Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning
We introduce Ego-R1, a novel framework for reasoning over ultra-long (i.e.,
in days and weeks) egocentric videos, which leverages a structured
Chain-of-Tool-Thought (CoTT) process, orchestrated by an Ego-R1 Agent trained
via reinforcement learning (RL). Inspired by human problem-solving strategies,
CoTT decomposes complex reasoning into modular steps, with the RL agent
invoking specific tools, one per step, to iteratively and collaboratively
answer sub-questions tackling such tasks as temporal retrieval and multi-modal
understanding. We design a two-stage training paradigm involving supervised
finetuning (SFT) of a pretrained language model using CoTT data and RL to
enable our agent to dynamically propose step-by-step tools for long-range
reasoning. To facilitate training, we construct a dataset called Ego-R1 Data,
which consists of Ego-CoTT-25K for SFT and Ego-QA-4.4K for RL. Furthermore, our
Ego-R1 agent is evaluated on a newly curated week-long video QA benchmark,
Ego-R1 Bench, which contains human-verified QA pairs from hybrid sources.
Extensive results demonstrate that the dynamic, tool-augmented chain-of-thought
reasoning by our Ego-R1 Agent can effectively tackle the unique challenges of
understanding ultra-long egocentric videos, significantly extending the time
coverage from few hours to a week.
Ego-R1, a reinforcement learning-based framework, uses a structured tool-augmented chain-of-thought process to reason over ultra-long egocentric videos, achieving better performance than existing methods by extending time coverage to a week.
Chain-of-Tool-Thought CoTT reinforcement learning RL pretrained language model supervised finetuning SFT Ego-CoTT-25K Ego-QA-4.4K Ego-R1 Bench video QA temporal retrieval multi-modal understanding