And it looks like GPT-Neo just added itself to the group of bfloat16-pretrained models.
  
      github.com/huggingface/transformers
  
  
    
  
	  
  

  
    
      FP16 overflow with GPT-Neo when using sequence lengths of 2048. 40
    

    
      
        opened 
        
          
        
        Apr 6, 2021
      
      


      
        
          
          LouisCastricato
        
      
    
  



  Environment info
transformers version: 4.5.0.dev0
Platform: Linux-5.4.0-54-generic-x86_64-with-glibc2.29
Python version: 3.8.5
PyTorch version (GPU?): 1.8.0+cu111
Tensorflow version (GPU?): N/A
Using GPU in script?: Yes
Using distributed or parallel set-up...