Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems
<|reference_start|>Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems: Continual Learning (CL) involves fine-tuning pre-trained models with new data while maintaining the performance on the pre-trained data. This is particularly relevant for expanding multilingual ASR (MASR) capabilities. However, existing CL methods, mainly designed for computer vision and reinforcement learning tasks, often yield sub-optimal results when directly applied to MASR. We hypothesise that this is because CL of the auto-regressive decoder in the MASR model is difficult. To verify this, we propose four optimizations on the decoder. They include decoder-layer gradient surgery, freezing unused token embeddings, suppressing output of newly added tokens, and learning rate re-scaling. Our experiments on adapting Whisper to 10 unseen languages from the Common Voice dataset demonstrate that these optimizations reduce the Average Word Error Rate (AWER) of pretrained languages from 14.2% to 12.4% compared with Experience Replay, without compromising the AWER of new languages.<|reference_end|>
@article{kwok2024continual,
    title={Continual Learning Optimizations for Auto-regressive Decoder of
  Multilingual ASR systems},
    author={Chin Yuen Kwok, Jia Qi Yip, Eng Siong Chng},
    journal={arXiv preprint arXiv:2407.03645},
    year={2024},
    doi={10.21437/Interspeech.2024-205},
    archivePrefix={arXiv},
    eprint={2407.03645},
    primaryClass={cs.CL cs.SD eess.AS}
}