Based on your code it seems you are using alban’s 3rd approach, which uses more memory and is slower than the other approaches, since it’s accumulating the computation graphs in each iteration and cannot free the intermediate tensors.
If you want to save memory, I would recommend to try out the 2nd approach.