The issue is mostly resolved here

  
      github.com/pytorch/xla
  
  
    
  
	  
  

  
    
      Problem with model accuracy (after restore) on TPU 41
    

    
      
        opened 
        
          
        
        Oct 27, 2019
      
      


      
        
          
          mgrankin
        
      
    
  



  🐛 Bug
I’m training GPT-2 from huggingface/transformers on TPU. It’s training well. At the end of a training I’ve got loss around...