I tried to repro this with a minimal example:
import torch

tensor = torch.randn(32768, 32768)
for i in range(tensor.size(1)):
    slice = tensor[:,i].detach().clone()
    torch.save(slice, 'asdf')

But I only see constant memory usage of about ~4GiB here, which is what we would expect from 2**30 elements * 4 bytes per element.