Comparative Analysis of Modality Fusion Approaches for Audio-Visual Person Identification and Verification
<|reference_start|>Comparative Analysis of Modality Fusion Approaches for Audio-Visual Person Identification and Verification: Multimodal learning involves integrating information from various modalities to enhance learning and comprehension. We compare three modality fusion strategies in person identification and verification by processing two modalities: voice and face. In this paper, a one-dimensional convolutional neural network is employed for x-vector extraction from voice, while the pre-trained VGGFace2 network and transfer learning are utilized for face modality. In addition, gammatonegram is used as speech representation in engagement with the Darknet19 pre-trained network. The proposed systems are evaluated using the K-fold cross-validation technique on the 118 speakers of the test set of the VoxCeleb2 dataset. The comparative evaluations are done for single-modality and three proposed multimodal strategies in equal situations. Results demonstrate that the feature fusion strategy of gammatonegram and facial features achieves the highest performance, with an accuracy of 98.37% in the person identification task. However, concatenating facial features with the x-vector reaches 0.62% for EER in verification tasks.<|reference_end|>
@article{farhadipour2024comparative,
    title={Comparative Analysis of Modality Fusion Approaches for Audio-Visual
  Person Identification and Verification},
    author={Aref Farhadipour, Masoumeh Chapariniya, Teodora Vukovic, Volker Dellwo},
    journal={arXiv preprint arXiv:2409.00562},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.00562},
    primaryClass={eess.AS cs.CV cs.MM cs.SD}
}