A dual task learning approach to fine-tune a multilingual semantic speech encoder for Spoken Language Understanding
<|reference_start|>A dual task learning approach to fine-tune a multilingual semantic speech encoder for Spoken Language Understanding: Self-Supervised Learning is vastly used to efficiently represent speech for Spoken Language Understanding, gradually replacing conventional approaches. Meanwhile, textual SSL models are proposed to encode language-agnostic semantics. SAMU-XLSR framework employed this semantic information to enrich multilingual speech representations. A recent study investigated SAMU-XLSR in-domain semantic enrichment by specializing it on downstream transcriptions, leading to state-of-the-art results on a challenging SLU task. This study's interest lies in the loss of multilingual performances and lack of specific-semantics training induced by such specialization in close languages without any SLU implication. We also consider SAMU-XLSR's loss of initial cross-lingual abilities due to a separate SLU fine-tuning. Therefore, this paper proposes a dual task learning approach to improve SAMU-XLSR semantic enrichment while considering distant languages for multilingual and language portability experiments.<|reference_end|>
@article{laperrière2024a,
    title={A dual task learning approach to fine-tune a multilingual semantic
  speech encoder for Spoken Language Understanding},
    author={Ga"elle Laperri`ere, Sahar Ghannay, Bassam Jabaian, Yannick Est`eve},
    journal={arXiv preprint arXiv:2406.12141},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.12141},
    primaryClass={cs.CL cs.SD eess.AS}
}