Can you try normalizing your input images at the test time, before making predictions? (since the images are also normalized during training)
norm_image = transforms.Normalize(0.3812, 0.1123)
image = norm_image(image)