Inconsistency-Aware Cross-Attention for Audio-Visual Fusion in Dimensional Emotion Recognition
<|reference_start|>Inconsistency-Aware Cross-Attention for Audio-Visual Fusion in Dimensional Emotion Recognition: Leveraging complementary relationships across modalities has recently drawn a lot of attention in multimodal emotion recognition. Most of the existing approaches explored cross-attention to capture the complementary relationships across the modalities. However, the modalities may also exhibit weak complementary relationships, which may deteriorate the cross-attended features, resulting in poor multimodal feature representations. To address this problem, we propose Inconsistency-Aware Cross-Attention (IACA), which can adaptively select the most relevant features on-the-fly based on the strong or weak complementary relationships across audio and visual modalities. Specifically, we design a two-stage gating mechanism that can adaptively select the appropriate relevant features to deal with weak complementary relationships. Extensive experiments are conducted on the challenging Aff-Wild2 dataset to show the robustness of the proposed model.<|reference_end|>
@article{rajasekhar2024inconsistency-aware,
    title={Inconsistency-Aware Cross-Attention for Audio-Visual Fusion in
  Dimensional Emotion Recognition},
    author={G Rajasekhar, Jahangir Alam},
    journal={arXiv preprint arXiv:2405.12853},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.12853},
    primaryClass={cs.CV}
}