Yes.
Because the allocator from cuda is a bit slow, we have a custom one in pytorch that is much faster.
So when you free a Tensor, its memory is returned to the custom allocator that can then reuse it for other Tensors.