Solved by alex.veuthey in post #4 
              
              
                I don’t know much about PyTorch’s internal behaviour, but I’ve never observed this. Even if you specifically load the previous model in the code, it doesn’t explain the big change in initial performance. 
One thing that could explain it is that your dataset has very easy samples and very hard sample…