Lai et al. [280] iteratively prune and subsequently fine-tune wav2vec2.0 on downstream tasks
to obtained improved results over fine-tuned wav2vec2.0. Winata et al. [593] employ low-rank
transformers to excise the model size by half and increase the inference speed by 1.35 times. Peng
et al. [422] employ KD and quantization to make wav2vec2.0 twice as fast, twice as energy efficient,
and 4.8 times smaller at the cost of a 7% increase in WER. Without the KD step, the model is 3.6
times smaller with mere 0.1% WER degradation.
AReviewofDeepLearningTechniquesforSpeechProcessing