So, I have written a dedicated prg to load the train & test sets used to to train and test the ResNet20 model where all the BN s where tracking stats as mentioned in my post of 10th December. Then I load the trained model and alternatively process a batch of training samples setting model.train() an a batch of testing samples setting model.eval(). Notice that here I  do not use (with torch.no_grad()) as I am using the same code for both processing mode, I just switch the model.train/eval at the beginning of the processing.
So far so good, here are the results which I should say puzzled me:

Use device…:  cuda
train_loader length= 781
test_loader length= 781
batch size:  128
train[ 0 ]
out layer3:  tensor(0., device=‘cuda:0’, grad_fn=) tensor(298.8851, device=‘cuda:0’, grad_fn=) tensor(0.2251, device=‘cuda:0’, grad_fn=) tensor(0.8224, device=‘cuda:0’, grad_fn=)
Train : loss =  6.8468918800354
test [ 0 ]
out layer3:  tensor(0., device=‘cuda:0’, grad_fn=) tensor(196.5387, device=‘cuda:0’, grad_fn=) tensor(0.2451, device=‘cuda:0’, grad_fn=) tensor(0.8113, device=‘cuda:0’, grad_fn=)
Test  : loss =  7.388835430145264
train[ 1 ]
out layer3:  tensor(0., device=‘cuda:0’, grad_fn=) tensor(532.2750, device=‘cuda:0’, grad_fn=) tensor(0.2222, device=‘cuda:0’, grad_fn=) tensor(0.7947, device=‘cuda:0’, grad_fn=)
Train : loss =  7.686824798583984
test [ 1 ]
out layer3:  tensor(0., device=‘cuda:0’, grad_fn=) tensor(235.0206, device=‘cuda:0’, grad_fn=) tensor(0.2438, device=‘cuda:0’, grad_fn=) tensor(0.7707, device=‘cuda:0’, grad_fn=)
Test  : loss =  7.462079048156738
train[ 2 ]
out layer3:  tensor(0., device=‘cuda:0’, grad_fn=) tensor(148.6887, device=‘cuda:0’, grad_fn=) tensor(0.2305, device=‘cuda:0’, grad_fn=) tensor(0.8213, device=‘cuda:0’, grad_fn=)
Train : loss =  7.347263813018799
test [ 2 ]
out layer3:  tensor(0., device=‘cuda:0’, grad_fn=) tensor(78.3936, device=‘cuda:0’, grad_fn=) tensor(0.2437, device=‘cuda:0’, grad_fn=) tensor(0.6861, device=‘cuda:0’, grad_fn=)
Test  : loss =  7.019978046417236
train[ 3 ]
out layer3:  tensor(0., device=‘cuda:0’, grad_fn=) tensor(386.5198, device=‘cuda:0’, grad_fn=) tensor(0.2214, device=‘cuda:0’, grad_fn=) tensor(0.8807, device=‘cuda:0’, grad_fn=)
Train : loss =  6.6856513023376465
test [ 3 ]
out layer3:  tensor(0., device=‘cuda:0’, grad_fn=) tensor(124.2858, device=‘cuda:0’, grad_fn=) tensor(0.2424, device=‘cuda:0’, grad_fn=) tensor(0.7225, device=‘cuda:0’, grad_fn=)
Test  : loss =  7.320082187652588
train[ 4 ]
out layer3:  tensor(0., device=‘cuda:0’, grad_fn=) tensor(289.6568, device=‘cuda:0’, grad_fn=) tensor(0.2118, device=‘cuda:0’, grad_fn=) tensor(0.7750, device=‘cuda:0’, grad_fn=)
Train : loss =  6.753640651702881
test [ 4 ]
out layer3:  tensor(0., device=‘cuda:0’, grad_fn=) tensor(118.0012, device=‘cuda:0’, grad_fn=) tensor(0.2386, device=‘cuda:0’, grad_fn=) tensor(0.6660, device=‘cuda:0’, grad_fn=)
Test  : loss =  6.969400405883789

Well, what is puzzling me is that I was expecting a much lower loss for the “train batches” then the “test batches”, and I far as I can see there are no big difference between the mode (model.train()) and (mode.test()). May be I am wrong as I should have performed a regular training with optimization, gradient descent and so on and print the stats of the output of the BN layers ???