VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding
<|reference_start|>VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding: We explore how reconciling several foundation models (large language models and vision-language models) with a novel unified memory mechanism could tackle the challenging video understanding problem, especially capturing the long-term temporal relations in lengthy videos. In particular, the proposed multimodal agent VideoAgent: 1) constructs a structured memory to store both the generic temporal event descriptions and object-centric tracking states of the video; 2) given an input task query, it employs tools including video segment localization and object memory querying along with other visual foundation models to interactively solve the task, utilizing the zero-shot tool-use ability of LLMs. VideoAgent demonstrates impressive performances on several long-horizon video understanding benchmarks, an average increase of 6.6% on NExT-QA and 26.0% on EgoSchema over baselines, closing the gap between open-sourced models and private counterparts including Gemini 1.5 Pro.<|reference_end|>
@article{fan2024videoagent:,
    title={VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding},
    author={Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, Qing Li},
    journal={arXiv preprint arXiv:2403.11481},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.11481},
    primaryClass={cs.CV}
}