The fact that the allocated memory is the same means that we use exactely the same amount of memory for Tensors. So that is good.
What is not tracked in this is:

cuda driver internal memory
cudnn buffers

To check the first, you can see how much cuda memory is used when you only do:
import torch
a = torch.rand(1).cuda()

To check the second, it is a bit harder. You can try to set torch.backends.deterministic = True. This forces to use the default deterministic algorithm that is usually not too memory hungry.