I have found something but really do not understand :<)

If I use at the beginning of training session (notice that the reference sample inputs are filled with ones):

torch.backends.cudnn.deterministic =True
torch.backends.cudnn.benchmark = False

and at the debugging session the same variable settings then
max abs error:  0.0

while if the debugging session  starts with
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.benchmark = True

then
max abs error:  9.5367431640625e-07


If now the training session is done with

torch.backends.cudnn.deterministic = False
torch.backends.cudnn.benchmark = True

the debugging session answer is independent of the torch.backends.cudnn.deterministic = False/True and
torch.backends.cudnn.benchmark = True/False values, and it yields
max abs error:  4.76837158203125e-07


I have also used the torch.backends.cudnn.enabled flag in the debugging session (the two other flags are commented):

torch.backends.cudnn.enabled = True
#torch.backends.cudnn.deterministic = True
#torch.backends.cudnn.benchmark = False


then
max abs error:  0.0

While if
torch.backends.cudnn.enabled = False

then
max abs error:  2.384185791015625e-06

So, it is clear that the  torch.backends.cudnn flags matter to recover the results after model reloading when the model is at least of ResNet architecture, may be due to the BatchNorm layers. But, then does it means that the arithmetic floating precision is the origin of all my troubles with BN??? What then should be the correct settings to be used?
Just in case you want more infos on my system:
PyTorch version: 1.3.1
Is debug build: No
CUDA used to build PyTorch: 10.1.243
OS: CentOS Linux release 7.7.1908 (Core)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
CMake version: version 2.8.12.2
Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.105
GPU models and configuration: GPU 0: Tesla V100-PCIE-32GB
Nvidia driver version: 418.87.01
cuDNN version: /opt/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.5.0
Versions of relevant libraries:
[pip] numpy==1.17.4
[pip] numpydoc==0.9.1
[pip] torch==1.3.1
[pip] torchvision==0.4.2
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.4                      243
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.0.15           py37ha843d7b_0
[conda] mkl_random                1.1.0            py37hd6b4f25_0
[conda] pytorch                   1.3.1           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchvision               0.4.2                py37_cu101    pytorch