Solved by melgor in post #4 
              
              
                Maybe it’s issue connected with Loss-scaling? 
First of all, read this blog post from NVIDIA 
The pytorch code for scaling loss is here 
Also make sure that input to network is CudaHalf. 
Let us know if this helped.