Towards Multimodal Open-Set Domain Generalization and Adaptation through Self-supervision
<|reference_start|>Towards Multimodal Open-Set Domain Generalization and Adaptation through Self-supervision: The task of open-set domain generalization (OSDG) involves recognizing novel classes within unseen domains, which becomes more challenging with multiple modalities as input. Existing works have only addressed unimodal OSDG within the meta-learning framework, without considering multimodal scenarios. In this work, we introduce a novel approach to address Multimodal Open-Set Domain Generalization (MM-OSDG) for the first time, utilizing self-supervision. To this end, we introduce two innovative multimodal self-supervised pretext tasks: Masked Cross-modal Translation and Multimodal Jigsaw Puzzles. These tasks facilitate the learning of multimodal representative features, thereby enhancing generalization and open-class detection capabilities. Additionally, we propose a novel entropy weighting mechanism to balance the loss across different modalities. Furthermore, we extend our approach to tackle also the Multimodal Open-Set Domain Adaptation (MM-OSDA) problem, especially in scenarios where unlabeled data from the target domain is available. Extensive experiments conducted under MM-OSDG, MM-OSDA, and Multimodal Closed-Set DG settings on the EPIC-Kitchens and HAC datasets demonstrate the efficacy and versatility of the proposed approach. Our source code is available at https://github.com/donghao51/MOOSA.<|reference_end|>
@article{dong2024towards,
    title={Towards Multimodal Open-Set Domain Generalization and Adaptation through
  Self-supervision},
    author={Hao Dong, Eleni Chatzi, Olga Fink},
    journal={arXiv preprint arXiv:2407.01518},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.01518},
    primaryClass={cs.CV cs.AI cs.LG}
}