Hi Alban,
sorry I should have been more specific. If I try to train a model that includes this normalization instead of the pytorch implementation it will not train properly (loss does not go down).
If I do not deatch() mn and sd then the loss is nan (which I don’t understand because I am adding +1e-8 to sd to prevent just that.
Regards,
Fabian