Solved by ptrblck in post #2 
              
              
                Yes, the dtype would be recorded by Autograd and the backward computation would be executed in the same precision as its forward part, if you don’t change it. 
This would also mean that your FP16 gradient updates might easily underflow, which is why we recommend to use the mixed-precision training u…