Hi, have a look at this post/library. Using half precision is not straight forward

  
      
      GitHub
  
  
    

NVIDIA/apex 96

A PyTorch Extension:  Tools for easy mixed precision and distributed training in Pytorch - NVIDIA/apex


  
  
    
    
  
  



  
    
    
    Training partially with fp16 autograd
  
  
    Hi! 
I was wondering if it’s possible to train partially with fp16. 
I’m dealing with a very big network. I would like to use fp16 in a submodule of the network. 
Does autograd deal with casting from float16 to float32 during the backprop? 
In other post i saw someone who recommends to use fp32 on batch normalization not to have convergence issues. So there should be no problem right?