hi @zb1 the official examples include  Roberta distillation using the DistillBert method. You can find it here

  
      
      GitHub
  
  
    

huggingface/transformers 28

🤗Transformers: State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0. - huggingface/transformers