Solved by ptrblck in post #2 
              
              
                torch.cuda.empty_cache() will, as the name suggests, empty the reusable GPU memory cache. PyTorch uses a custom memory allocator, which reuses freed memory, to avoid expensive and synchronizing cudaMalloc calls. 
Since you are freeing this cache, PyTorch needs to reallocate the memory for each new d…