Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights from FairCVdb
<|reference_start|>Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights from FairCVdb: Despite the large body of work on fairness-aware learning for individual modalities like tabular data, images, and text, less work has been done on multimodal data, which fuses various modalities for a comprehensive analysis. In this work, we investigate the fairness and bias implications of multimodal fusion techniques in the context of multimodal AI-based recruitment systems using the FairCVdb dataset. Our results show that early-fusion closely matches the ground truth for both demographics, achieving the lowest MAEs by integrating each modality's unique characteristics. In contrast, late-fusion leads to highly generalized mean scores and higher MAEs. Our findings emphasise the significant potential of early-fusion for accurate and fair applications, even in the presence of demographic biases, compared to late-fusion. Future research could explore alternative fusion strategies and incorporate modality-related fairness constraints to improve fairness. For code and additional insights, visit: https://github.com/Swati17293/Multimodal-AI-Based-Recruitment-FairCVdb<|reference_end|>
@article{swati2024exploring,
    title={Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights
  from FairCVdb},
    author={Swati Swati, Arjun Roy and Eirini Ntoutsi},
    journal={arXiv preprint arXiv:2407.16892},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.16892},
    primaryClass={cs.CY cs.CL cs.CV cs.LG}
}