So memory fragmentation (which is likely the more precise than memory leak to describe the situation) is a thing with the caching allocator. If you wanted, you could set PYTORCH_NO_CUDA_MEMORY_CACHING=1 to get around this at the expense of doing all allocations/deallocations through cuda.