EAGLE: Egocentric AGgregated Language-video Engine
<|reference_start|>EAGLE: Egocentric AGgregated Language-video Engine: The rapid evolution of egocentric video analysis brings new insights into understanding human activities and intentions from a first-person perspective. Despite this progress, the fragmentation in tasks like action recognition, procedure learning, and moment retrieval, \etc, coupled with inconsistent annotations and isolated model development, hinders a holistic interpretation of video content. In response, we introduce the EAGLE (Egocentric AGgregated Language-video Engine) model and the EAGLE-400K dataset to provide a unified framework that integrates various egocentric video understanding tasks. EAGLE-400K, the \textit{first} large-scale instruction-tuning dataset tailored for egocentric video, features 400K diverse samples to enhance a broad spectrum of tasks from activity recognition to procedure knowledge learning. Moreover, EAGLE, a strong video multimodal large language model (MLLM), is designed to effectively capture both spatial and temporal information. In addition, we propose a set of evaluation metrics designed to facilitate a thorough assessment of MLLM for egocentric video understanding. Our extensive experiments demonstrate EAGLE's superior performance over existing models, highlighting its ability to balance task-specific understanding with holistic video interpretation. With EAGLE, we aim to pave the way for research opportunities and practical applications in real-world scenarios.<|reference_end|>
@article{bi2024eagle:,
    title={EAGLE: Egocentric AGgregated Language-video Engine},
    author={Jing Bi, Yunlong Tang, Luchuan Song, Ali Vosoughi, Nguyen Nguyen, and
  Chenliang Xu},
    journal={arXiv preprint arXiv:2409.17523},
    year={2024},
    doi={10.1145/3664647.3681618},
    archivePrefix={arXiv},
    eprint={2409.17523},
    primaryClass={cs.CV cs.AI}
}