I cannot reproduce the issue and get:
out = torch.randn(100,100)
out.cuda()
print(torch.cuda.memory_allocated(0))
> 0

out1 = torch.randn(100,100)
print(torch.cuda.memory_allocated(0))
> 0

Are you using a REPL environment where the output of the last operation might still be stored in the _ object?