Did you see the same effect during training by observing the training and validation loss (which would correspond to calling model.train() and model.eval()) or are you only seeing this issue after loading the trained model?