Maybe you can remove this normalization to see if you are still having these inf values.
If yes, this means the source of error is somewhere else than Normalize.
If not, could you try a dummy normalization with
normalize = torchvision.transforms.Normalize(mean=(1, 1, 1), std=(1, 1, 1) )
?