Found a 10x memory increase with native fp16 (pt1.6) - no such problem with nvidia-apex (pt1.5)

  
      github.com/huggingface/transformers
  
  
    
  
	  
  

  
    
      [s2s finetune] huge increase in memory demands with --fp16 76
    

    
      
        opened 
        
          
        
        Nov 8, 2020
      
      


      
        
          
          stas00
        
      
    
  



  While working on #8353 I discovered that --fp16 causes a 10x+ increase in gpu memory demands.
e.g. I can run bs=12 w/o...