When the encoder is reduced to 16-layers, the parameter count decreases by an additional 19%.
However, the OOD WER performance degrades by 3.1% absolute. This suggests that having a deep
encoder is paramount for maintaining strong WER performance of distilled Seq2Seq ASR models.
DISTIL-WHISPER