Ego-VPA: Egocentric Video Understanding with Parameter-efficient Adaptation
<|reference_start|>Ego-VPA: Egocentric Video Understanding with Parameter-efficient Adaptation: Video understanding typically requires fine-tuning the large backbone when adapting to new domains. In this paper, we leverage the egocentric video foundation models (Ego-VFMs) based on video-language pre-training and propose a parameter-efficient adaptation for egocentric video tasks, namely Ego-VPA. It employs a local sparse approximation for each video frame/text feature using the basis prompts, and the selected basis prompts are used to synthesize video/text prompts. Since the basis prompts are shared across frames and modalities, it models context fusion and cross-modal transfer in an efficient fashion. Experiments show that Ego-VPA excels in lightweight adaptation (with only 0.84% learnable parameters), largely improving over baselines and reaching the performance of full fine-tuning.<|reference_end|>
@article{wu2024ego-vpa:,
    title={Ego-VPA: Egocentric Video Understanding with Parameter-efficient
  Adaptation},
    author={Tz-Ying Wu, Kyle Min, Subarna Tripathi, Nuno Vasconcelos},
    journal={arXiv preprint arXiv:2407.19520},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.19520},
    primaryClass={cs.CV cs.LG}
}