Interesting… The errors seems to be that some gradients are not floating point numbers.
Do you have a custom autograd Function in your code?
If not, can you share a small code sample (30 lines) that reproduces the issue)?