Hi,
(I use pytorch 1.3.1, and use a GPU V100)
Here is my problem (new one): I have a model with Batch Normalisation. During the training and testing phase (same script), at each epoch, I use  model.train() before entering a loop on training batch sets to perform optimization and model.eval() before evaluating the current model. Also, at each epoch, I save the model state dictionnary as well as the optimizer, scheduler as
     state = { 
          'epoch': epoch + 1,
           'model_state_dict': model.state_dict(),
           'scheduler_state_dict': scheduler.state_dict(),
           'optimizer_state_dict': optimizer.state_dict()
      }
 torch.save(state,"model.pth")

Notice that before this saving phase, as I have performed a testing I have set model.eval().
I have recorded during this training/testing phase the statistics of the ouputs of the 3 last BN layers. Typically from the last epoch during the training  I got for (min, max, mean, std)
last bn2:  -46.39485549926758 36.850852966308594 -0.39066851139068604 1.3487299680709839
last bn2:  -28.653053283691406 15.431465148925781 -0.6616867780685425 1.2987738847732544
last bn2:  -47.44236373901367 19.981163024902344 -0.9964369535446167 1.3029693365097046

and the testing
last bn2:  -58.15468215942383 40.56957244873047 -0.29340481758117676 1.1473218202590942
last bn2:  -21.610492706298828 14.811783790588379 -0.47804850339889526 1.0458149909973145
last bn2:  -70.31809997558594 41.03254318237305 -0.8271327018737793 1.351212978363037

(notice that these layers are followed by ReLU activation).
Ok so far so good. Now, in an other script, I load the model and set it to model.eval(),
model.to(device)
checkpoint = torch.load(checkpoint_file)
model.load_state_dict(checkpoint['model_state_dict'])

Then on the same sets (both train and test) as used in the previous script, I process some samples and got
train[ 9 ]
last bn2:  -365.64263916015625 177.18020629882812 -0.2252720594406128 1.798872470855713
last bn2:  -135.1555938720703 49.426048278808594 -1.0530227422714233 1.5416172742843628
last bn2:  -89.00902557373047 36.76680374145508 -0.7778999209403992 0.9938169121742249

for a training sample and
test [ 9 ]
last bn2:  -240.713134765625 114.61311340332031 -0.19601963460445404 1.3636577129364014
last bn2:  -108.94783782958984 55.55070114135742 -1.0378086566925049 1.46784508228302
last bn2:  -100.44957733154297 29.672212600708008 -0.7665001153945923 0.9042347073554993

for a testing sample.
As u can see the stats are completely different !
Can you give me some hints to understand why there is this difference eg.:

bad saving procedure
bad loading procedure
aogr (Any other good reason)

Thanks