ptrblck:

Could you check the stats (min, max, mean, median, shape) or I_features_i and T_features_i , which cause the NaNs?
If possible, could you upload them so that we could use them to debug?


I don’t know if this works, but here is the output from the first few iterations: https://pastebin.com/YgH70rth 4
I also included the gradscaler statedict output, in case it is useful. I wouldn’t know what to say about the stats, but there are no NaNs. The values are small, but I don’t know if that could cause problems? Its at that convolution where the problem is happening.