PyTorch uses a caching allocator and reused already allocated memory.
I think the code tries to focus on readability of the code and as you said, these small perf. gains are sometimes not possible without bad hacks.