That’s a strange behavior. Could you explain a bit more about your model?
Have you changed the momentum in BatchNorm?
Are you preprocessing the data differently?
By default, Modules are initialized with training=True (code 28).