Momentum was 0.1 and eps was 1e-5.
The negative var values appear during the training so I guess that even with the regular fp16 training procedure there are still ways to have an overflow here.
After I changed my model I have more stable trainings. I torch.clamp the output and the input of the model, I also apply tanh after each convolution such that having big values here or in the batchnorm would be useless.
Apart from an overflow I don’t see how I could get a negative variance in the batchnorm