Hi,
From playing a bit to try and do better than the custom GPU allocator that pytorch is using, it is quite hard to do runtime-wise !
Basically, this 1GB buffer will be reused in the next iteration when you need 1GB again, and so it will not use more memory and the allocation will be super fast. So the fact that torch outputs a new 1GB tensor is not a problem at all, this buffer will be needed anyway if you use the autograd for temporary results.
The only possible problem that the allocator could create is a total memory usage higher than the memory needed for all your tensors (because it creates some holes in the memory). In that case, it is possible that a workload that should use 11.8GB of memory does not actually fit on a gpu with 12GB of memory. But in that case, I would say that you will loose less time by just reducing your batch size (or splitting your batch in two and do two forward backward for a single parameters update) than if you’re playing with freeing the cache of the allocator or deleting Tensors by hand.