Solved by ptrblck in post #2 
              
              
                PyTorch uses a caching allocator and reused already allocated memory. 
I think the code tries to focus on readability of the code and as you said, these small perf. gains are sometimes not possible without bad hacks.