I’m not sure if you were seeing the increased GPU memory usage in each iteration of the training and where toep is initialized. It would be expected to see an increased GPU memory usage in this loop alone:
for j in range(len(toep)):
    toep[j] = toep[j].cuda()

since you are moving data to the device.
However, removing toep should release the memory again as seen here:
toep = [torch.randn(1024) for _ in range(1024)]

print(torch.cuda.memory_allocated()/1024**2)
# > 0.0

for j in range(len(toep)):
    toep[j] = toep[j].cuda()
    
# 1024 * 1024 * 4 = 4MB expected
print(torch.cuda.memory_allocated()/1024**2)
# > 4.0

# free it
del toep
print(torch.cuda.memory_allocated()/1024**2)
# > 0.0

Are you increasing the length of toep somehow or are storing references to the tensors somewhere else, which could disallow their deletion?