Transferable-guided Attention Is All You Need for Video Domain Adaptation
<|reference_start|>Transferable-guided Attention Is All You Need for Video Domain Adaptation: Unsupervised domain adaptation (UDA) in videos is a challenging task that remains not well explored compared to image-based UDA techniques. Although vision transformers (ViT) achieve state-of-the-art performance in many computer vision tasks, their use in video UDA has been little explored. Our key idea is to use transformer layers as a feature encoder and incorporate spatial and temporal transferability relationships into the attention mechanism. A Transferable-guided Attention (TransferAttn) framework is then developed to exploit the capacity of the transformer to adapt cross-domain knowledge across different backbones. To improve the transferability of ViT, we introduce a novel and effective module, named Domain Transferable-guided Attention Block (DTAB). DTAB compels ViT to focus on the spatio-temporal transferability relationship among video frames by changing the self-attention mechanism to a transferability attention mechanism. Extensive experiments were conducted on UCF-HMDB, Kinetics-Gameplay, and Kinetics-NEC Drone datasets, with different backbones, like ResNet101, I3D, and STAM, to verify the effectiveness of TransferAttn compared with state-of-the-art approaches. Also, we demonstrate that DTAB yields performance gains when applied to other state-of-the-art transformer-based UDA methods from both video and image domains. Our code is available at https://github.com/Andre-Sacilotti/transferattn-project-code.<|reference_end|>
@article{sacilotti2024transferable-guided,
    title={Transferable-guided Attention Is All You Need for Video Domain
  Adaptation},
    author={Andr'e Sacilotti, Samuel Felipe dos Santos, Nicu Sebe, Jurandy
  Almeida},
    journal={IEEE/CVF Winter Conference on Applications of Computer Vision
  (WACV'25), 2025, pp. 1-11},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.01375},
    primaryClass={cs.CV}
}