Shouldn’t be the case

  
    
    
    Training partially with fp16 autograd
  
  
    Yes, it’s possible. 
However, using FP16, you should take care of some possible pitfalls, e.g. imprecise weight updates or gradient underflow. NVIDIA recently published their mixed precision utilities for PyTorch named apex. 
On the one side you have the automatic mixed precision training using amp, while on the other side the FP16_Optimizer gives you a bit more control over your workflow. 
Have a look at this small tutorial I’ve written a few days ago. It’s still not finished and @mcarilli is c…