Whisper-SV: Adapting Whisper for Low-data-resource Speaker Verification
<|reference_start|>Whisper-SV: Adapting Whisper for Low-data-resource Speaker Verification: Trained on 680,000 hours of massive speech data, Whisper is a multitasking, multilingual speech foundation model demonstrating superior performance in automatic speech recognition, translation, and language identification. However, its applicability in speaker verification (SV) tasks remains unexplored, particularly in low-data-resource scenarios where labeled speaker data in specific domains are limited. To fill this gap, we propose a lightweight adaptor framework to boost SV with Whisper, namely Whisper-SV. Given that Whisper is not specifically optimized for SV tasks, we introduce a representation selection module to quantify the speaker-specific characteristics contained in each layer of Whisper and select the top-k layers with prominent discriminative speaker features. To aggregate pivotal speaker-related features while diminishing non-speaker redundancies across the selected top-k distinct layers of Whisper, we design a multi-layer aggregation module in Whisper-SV to integrate multi-layer representations into a singular, compacted representation for SV. In the multi-layer aggregation module, we employ convolutional layers with shortcut connections among different layers to refine speaker characteristics derived from multi-layer representations from Whisper. In addition, an attention aggregation layer is used to reduce non-speaker interference and amplify speaker-specific cues for SV tasks. Finally, a simple classification module is used for speaker classification. Experiments on VoxCeleb1, FFSVC, and IMSV datasets demonstrate that Whisper-SV achieves EER/minDCF of 2.22%/0.307, 6.14%/0.488, and 7.50%/0.582, respectively, showing superior performance in low-data-resource SV scenarios.<|reference_end|>
@article{zhang2024whisper-sv:,
    title={Whisper-SV: Adapting Whisper for Low-data-resource Speaker Verification},
    author={Li Zhang, Ning Jiang, Qing Wang, Yue Li, Quan Lu, and Lei Xie},
    journal={arXiv preprint arXiv:2407.10048},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.10048},
    primaryClass={cs.SD eess.AS}
}