Hi
I am using pytorch within a chatbot training routine and I would like to get FP16’s advantages in GPU memory/speed.   My routine seems to work fine using FP32.  In switching it to FP16, my problem appears to be caused by the loss.backward()  pytorch routine.  For my optimizer to work, I need to use the argument create_graph = True in  backward.  When I do that with the model I am working with (for any scale factor on the loss that I’ve tried), backward gives NaN for some of the parameter gradients while others appear to have reasonable values.  However, when I use the argument create_graph = False to see how these gradients compare with the former case, there doesn’t appear to be any NaN generated and the gradients that weren’t NaN in the former case agree with what is computed here.  Since the create_graph is supposed to allow computation of higher order derivatives (but not the first derivative), I am confused why the create_graph = True argument is having this effect.  Any suggestions you can provide would be appreciated!