How do you check the memory?
PyTorch uses a custom allocator to speed up GPU allocations. So it is expected that the memory in nvidia-smi doesn’t go down. You can use torch.cuda.memory_allocated() to see the memory that is actually used by Tensors.