I observed similar GPU memory behavior.
When I test two different implementations of the same function as below,
where ‘A’ is cuda.Tensor.
def function1(A):
    B = A**2 - 2*A
    C = torch.sqrt(B)
    return C

def function2(A):
    return torch.sqrt(A**2 - 2*A)

Both functions are the same as a function.
However, function1 seems to assign GPU memory to local variables ‘B’, 'C’
on the other hand, function2 seems to only assign GPU memory to memory needed to calculate torch.sqrt(A**2 - 2*A)
which is presumably the same size as A
Thus, in terms of memory usage, it seems that function2 is twice efficient than function1.
This doesn’t apply to all the cases, but in many cases, removing intermediate variables reduces GPU memory usage a lot in my programs.
This seems to be that underlying CUDA does not free memory immediately after the moment that memory is not needed anymore.
I think some GPU memory Garbage Collection method in pytorch is needed for efficient GPU memory management.