Hi,
pytorch uses a custom gpu allocator that caches the old allocations.
This means that if you perform the same allocation/deallocation  at every iteration of your loop, the allocator will actually be very efficient at allocating this new memory.
I would expect to see no runtime difference between the code that does the allocation every time and the one that pre-allocate the tensor. So I would say it is not worth the complexity of the code.