I think the memory summary is confusing here. Are you seeing 4GB more used, or 4GB that is used and not available for allocations? A litmus test here would be to keep allocating tensors after the function returns and see how many succeed before an OOM error is thrown.