Dear @ptrblck,
I have proceeded to the following experiment:

I have switch off both Random Shuffling of the data_laoders as well as Data Augmentation

I have dumped the first 5 images input of the first 10 batches both Train set and Test sets during the Training session (1st script) and during the Debugging session (2nd script). I have fully checked that the images at the input of the network was the same epoch per epoch during the training, notably the last epoch where I have saved the model; and the same images were loaded during the Debugging session which load also the model state dictionary.

So, Here are the outputs of the models:

on top panels (Training session last epoch): outputs of the 5 first  images of the train batch 0 (left), outputs of the 5 first  images of the train batch 9 (middle), outputs of the 5 first  images of the test batch 0 (right)
on bottom panels there are the outputs of the same inputs of the top panels but during the debugging session



02870×527 155 KB

Clearly there is something strange during the Debugging session concerning the model parameters. An idea?
I have just cross-checked that if I dump all the layers weights/bias stat (min/max/mean/std) before saving (1st script during training session), then the same weights/bias stat of all the layers are loaded in the debugging session (2nd script).
So if everything is ok: what can leads to different outputs if I give the same inputs to a model that seems to have the same parameter loaded (at least the same stats as described above?). Note that in the debugging session I switch to model.eval().
Thanks