HCQA @ Ego4D EgoSchema Challenge 2024
<|reference_start|>HCQA @ Ego4D EgoSchema Challenge 2024: In this report, we present our champion solution for Ego4D EgoSchema Challenge in CVPR 2024. To deeply integrate the powerful egocentric captioning model and question reasoning model, we propose a novel Hierarchical Comprehension scheme for egocentric video Question Answering, named HCQA. It consists of three stages: Fine-grained Caption Generation, Context-driven Summarization, and Inference-guided Answering. Given a long-form video, HCQA captures local detailed visual information and global summarised visual information via Fine-grained Caption Generation and Context-driven Summarization, respectively. Then in Inference-guided Answering, HCQA utilizes this hierarchical information to reason and answer given question. On the EgoSchema blind test set, HCQA achieves 75% accuracy in answering over 5,000 human curated multiple-choice questions. Our code will be released at https://github.com/Hyu-Zhang/HCQA.<|reference_end|>
@article{zhang2024hcqa,
    title={HCQA @ Ego4D EgoSchema Challenge 2024},
    author={Haoyu Zhang, Yuquan Xie, Yisen Feng, Zaijing Li, Meng Liu, Liqiang Nie},
    journal={arXiv preprint arXiv:2406.15771},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.15771},
    primaryClass={cs.CV cs.AI}
}