Task-Adapter: Task-specific Adaptation of Image Models for Few-shot Action Recognition
<|reference_start|>Task-Adapter: Task-specific Adaptation of Image Models for Few-shot Action Recognition: Existing works in few-shot action recognition mostly fine-tune a pre-trained image model and design sophisticated temporal alignment modules at feature level. However, simply fully fine-tuning the pre-trained model could cause overfitting due to the scarcity of video samples. Additionally, we argue that the exploration of task-specific information is insufficient when relying solely on well extracted abstract features. In this work, we propose a simple but effective task-specific adaptation method (Task-Adapter) for few-shot action recognition. By introducing the proposed Task-Adapter into the last several layers of the backbone and keeping the parameters of the original pre-trained model frozen, we mitigate the overfitting problem caused by full fine-tuning and advance the task-specific mechanism into the process of feature extraction. In each Task-Adapter, we reuse the frozen self-attention layer to perform task-specific self-attention across different videos within the given task to capture both distinctive information among classes and shared information within classes, which facilitates task-specific adaptation and enhances subsequent metric measurement between the query feature and support prototypes. Experimental results consistently demonstrate the effectiveness of our proposed Task-Adapter on four standard few-shot action recognition datasets. Especially on temporal challenging SSv2 dataset, our method outperforms the state-of-the-art methods by a large margin.<|reference_end|>
@article{cao2024task-adapter:,
    title={Task-Adapter: Task-specific Adaptation of Image Models for Few-shot
  Action Recognition},
    author={Congqi Cao, Yueran Zhang, Yating Yu, Qinyi Lv, Lingtong Min, Yanning
  Zhang},
    journal={arXiv preprint arXiv:2408.00249},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.00249},
    primaryClass={cs.CV}
}