We recommend to use the native automatic mixed-precision training using a master build or the nightly binaries.
For your use case, you might have forgotten to unscale the gradients or might have manually updated the parameters with invalid gradients.
Have a look at the gradient penalty example 8 for an example use case of autograd.grad.