Definitively there is something strange
I have filled with 1 the reference samples
    img_test_ref = torch.ones(10,64,64,5) # NxHxWxC  (this will be transfomed NCHW in the test_ref() function
    ebv_test_ref = torch.ones(10,1)
    z_test_ref   = torch.ones(10,1)
    sample_test_ref = {'image':img_test_ref,'ebv':ebv_test_ref,'z':z_test_ref}

Then, after 2 epochs I have reloaded the model and the above 1-fllled tensors
max abs error:  12.013355255126953

The reference outputs:
ref_out
 tensor([[-1.3784, -0.3093,  0.4179,  ..., -0.5395, -1.2264, -0.2501],
        [-1.3784, -0.3093,  0.4179,  ..., -0.5395, -1.2264, -0.2501],
        [-1.3784, -0.3093,  0.4179,  ..., -0.5395, -1.2264, -0.2501],
        ...,
        [-1.3784, -0.3093,  0.4179,  ..., -0.5395, -1.2264, -0.2501],
        [-1.3784, -0.3093,  0.4179,  ..., -0.5395, -1.2264, -0.2501],
        [-1.3784, -0.3093,  0.4179,  ..., -0.5395, -1.2264, -0.2501]],
       device='cuda:0') 
 tensor([[5.3705e-06, 1.5643e-05, 3.2369e-05,  ..., 1.2427e-05, 6.2522e-06,
         1.6597e-05],
        [5.3705e-06, 1.5643e-05, 3.2369e-05,  ..., 1.2427e-05, 6.2522e-06,
         1.6597e-05],
        [5.3705e-06, 1.5643e-05, 3.2369e-05,  ..., 1.2427e-05, 6.2522e-06,
         1.6597e-05],
        ...,
        [5.3705e-06, 1.5643e-05, 3.2369e-05,  ..., 1.2427e-05, 6.2522e-06,
         1.6597e-05],
        [5.3705e-06, 1.5643e-05, 3.2369e-05,  ..., 1.2427e-05, 6.2522e-06,
         1.6597e-05],
        [5.3705e-06, 1.5643e-05, 3.2369e-05,  ..., 1.2427e-05, 6.2522e-06,
         1.6597e-05]], device='cuda:0')

While the new output reads:
new_out
 tensor([[ 0.4656, -3.6460, -1.1858,  ..., -2.8810, -1.0660, -5.4568],
        [ 0.4656, -3.6460, -1.1858,  ..., -2.8810, -1.0660, -5.4568],
        [ 0.4656, -3.6460, -1.1858,  ..., -2.8810, -1.0660, -5.4568],
        ...,
        [ 0.4656, -3.6460, -1.1858,  ..., -2.8810, -1.0660, -5.4568],
        [ 0.4656, -3.6460, -1.1858,  ..., -2.8810, -1.0660, -5.4568],
        [ 0.4656, -3.6460, -1.1858,  ..., -2.8810, -1.0660, -5.4568]],
       device='cuda:0') 
 tensor([[6.3085e-07, 1.0335e-08, 1.2099e-07,  ..., 2.2210e-08, 1.3639e-07,
         1.6899e-09],
        [6.3085e-07, 1.0335e-08, 1.2099e-07,  ..., 2.2210e-08, 1.3639e-07,
         1.6899e-09],
        [6.3085e-07, 1.0335e-08, 1.2099e-07,  ..., 2.2210e-08, 1.3639e-07,
         1.6899e-09],
        ...,
        [6.3085e-07, 1.0335e-08, 1.2099e-07,  ..., 2.2210e-08, 1.3639e-07,
         1.6899e-09],
        [6.3085e-07, 1.0335e-08, 1.2099e-07,  ..., 2.2210e-08, 1.3639e-07,
         1.6899e-09],
        [6.3085e-07, 1.0335e-08, 1.2099e-07,  ..., 2.2210e-08, 1.3639e-07,
         1.6899e-09]], device='cuda:0')