Thanks,
also from Google It seems that knowledge distillation is something that we need to handle with care:

  
      

      arXiv.org
  

  
    

Does Knowledge Distillation Really Work? 4

  Knowledge distillation is a popular technique for training a small student
network to emulate a larger teacher model, such as an ensemble of networks. We
show that while knowledge distillation can improve student generalization, it
does not typically...