EPD: Long-term Memory Extraction, Context-awared Planning and Multi-iteration Decision @ EgoPlan Challenge ICML 2024
<|reference_start|>EPD: Long-term Memory Extraction, Context-awared Planning and Multi-iteration Decision @ EgoPlan Challenge ICML 2024: In this technical report, we present our solution for the EgoPlan Challenge in ICML 2024. To address the real-world egocentric task planning problem, we introduce a novel planning framework which comprises three stages: long-term memory Extraction, context-awared Planning, and multi-iteration Decision, named EPD. Given the task goal, task progress, and current observation, the extraction model first extracts task-relevant memory information from the progress video, transforming the complex long video into summarized memory information. The planning model then combines the context of the memory information with fine-grained visual information from the current observation to predict the next action. Finally, through multi-iteration decision-making, the decision model comprehensively understands the task situation and current state to make the most realistic planning decision. On the EgoPlan-Test set, EPD achieves a planning accuracy of 53.85% over 1,584 egocentric task planning questions. We have made all codes available at https://github.com/Kkskkkskr/EPD .<|reference_end|>
@article{shi2024epd:,
    title={EPD: Long-term Memory Extraction, Context-awared Planning and
  Multi-iteration Decision @ EgoPlan Challenge ICML 2024},
    author={Letian Shi, Qi Lv, Xiang Deng, Liqiang Nie},
    journal={arXiv preprint arXiv:2407.19510},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.19510},
    primaryClass={cs.RO cs.CV}
}