Yes I am aware of this paper since it had come out during the same time.
Here’s another from MSFT Research in case you want to dive even further:

  

      Microsoft Research – 19 Jan 21
  

  
    

3 deep learning mysteries: Ensemble, knowledge- and self-distillation 2

  Microsoft and CMU researchers begin to unravel 3 mysteries in deep learning related to ensemble, knowledge distillation & self-distillation. Discover how their work leads to the first theoretical proof with empirical evidence for ensemble in deep...


  

  
    
    
  

  


There are also works that show how knowledge distillation may ignore the skewed portions of the dataset.
But business needs and objectives drive the carefulness I think. Knowledge distillation has the promise to account for both without incorporating too much technical debt or complications in the overall pipeline. Amongst three popular compression schemes (quantization, pruning, and distillation) distillation is my favourite.