Does you actually see OOM errors due to this behavior?
The reason I ask this is because the allocator in pytorch is built to be as fast as possible, not to reduce fragmentation. The “trick” that is used to reduce fragmentation is that when no more memory is available. All the un-used blocks as freed (cudaFreed) and then a new block is allocated (cudaMalloced). The cuda driver can reduce fragementation much better than we could by remapping some memory.
It’s possible that if only 2GB were available. Your process will still run just fine with a peak usage of 2 at the first iteration and 1 afterwards.