Check Do we really need to distill? Jont Loss is all we need.
  
      
      github.com
  
  
    

legacyai/tf-transformers 34


  State of the art faster Natural Language Processing in Tensorflow 2.0 .

  
  
    
    
  
  


This model is 4x smaller than DistillBERT and better than it by GLUE score of 4 points.