Hi,
I share something I known before with you, but I don’t know if it will help you.
What I tried as follows:
intermediate = torch.randn(1,3,3,3)
copy_intermediate = intermediate

print(id(intermediate))
print(id(copy_intermediate))
# the same id
print(intermediate is copy_intermediate))
# True

It seems that assignment operation makes variables point to the same address ( exclude mutable variables like Integer and String), so I think you could not worry about the memory consumption.
Did you encounter high GPU wastage using the code posted?