Yeah, I’m talking about launching a new experiment with the same script. It’s weird for me because I have observed this phenomenon many times. Sometimes the starting loss is not lower, but sometimes it is. The decrease is so significant that I don’t think the random weight initialization is the cause. The only thing that I can think of is that the Pytorch is saving some state after the last training session and loading it into the new one. But it seems impossible.