The inverse normalization should be
x = z*sigma + mean
= (z + mean/sigma) * sigma
= (z - (-mean/sigma)) / (1/sigma),
since the normalization process is actually z = (x - mean) / sigma if you look carefully at the documentation of transforms.Normalize. I tested my original code and it worked fine.