Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition
<|reference_start|>Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition: In video-based emotion recognition, audio and visual modalities are often expected to have a complementary relationship, which is widely explored using cross-attention. However, they may also exhibit weak complementary relationships, resulting in poor representations of audio-visual features, thus degrading the performance of the system. To address this issue, we propose Dynamic Cross-Attention (DCA) that can dynamically select cross-attended or unattended features on the fly based on their strong or weak complementary relationship with each other, respectively. Specifically, a simple yet efficient gating layer is designed to evaluate the contribution of the cross-attention mechanism and choose cross-attended features only when they exhibit a strong complementary relationship, otherwise unattended features. We evaluate the performance of the proposed approach on the challenging RECOLA and Aff-Wild2 datasets. We also compare the proposed approach with other variants of cross-attention and show that the proposed model consistently improves the performance on both datasets.<|reference_end|>
@article{praveen2024cross-attention,
    title={Cross-Attention is Not Always Needed: Dynamic Cross-Attention for
  Audio-Visual Dimensional Emotion Recognition},
    author={R. Gnana Praveen, Jahangir Alam},
    journal={arXiv preprint arXiv:2403.19554},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.19554},
    primaryClass={cs.CV}
}