Hi,
I’m trying to understand and solve a problem where my loss goes to nan. Information I have:

Fp16 training (autocast, scale().backward, unscale, clip_grad_norm, scaler.step, scaler.update, zerograd) diverges to Nan
I found the issue in a batchnorm layer during an fp32 inference
It goes: convolution2d > x > batchnorm2d > some feature maps are full of NaN

After checking in depth (manually recomputing batchnorm), I found that some of my batchnorm.running_var have negative values which isn’t supposed to happen (it produces nan when the batchnorm applies sqrt() on these values).
How could the values in running_var be negative?
Does it mean that there’s an issue with the scaler during the training?
Thanks to anyone with more information on this issue