6.3.2 Knowledge Distillation (KD)
Knowledge distillation involves training a smaller model to mimic the behavior of a larger and
more complex model. This can be done by training the smaller model to predict the outputs of the
larger model or, by using, the larger model’s hidden representations as input to the smaller model.
Knowledge distillation is effective in reducing the computational cost of training and inference.
Cho et al. [81] conducted knowledge distillation (KD) by directly applying it to the downstream
task. One way to improve this approach is to use KD as pre-training for various downstream tasks,
thus allowing for knowledge reuse. A noteworthy result achieved by Denisov and Vu [107] was
using KD in pretraining. However, they achieved this by initializing an utterance encoder with a
trained ASR model’s backbone, followed by a trained NLU backbone. Knowledge distillation can be
applied directly into a wav2vec 2.0 encoder without ASR training and a trained NLU module to
AReviewofDeepLearningTechniquesforSpeechProcessing