I don’t know much about PyTorch’s internal behaviour, but I’ve never observed this. Even if you specifically load the previous model in the code, it doesn’t explain the big change in initial performance.
One thing that could explain it is that your dataset has very easy samples and very hard samples, and your dataset is shuffled at each training session, therefore the starting batches could include only or mostly hard samples and in other cases only or mostly easy samples… Do you use shuffle in your DataLoader and could you try several experiments (just stop them after a few batches) without shuffle if you do? That could verify this hypothesis.