Hello,
In my shallow view, you could use temporary buffer to perform calculation and release it after accumulating. But I think if the variable requires gradient, the intermediate computation graph is needed in each iteration and it is unavoidable.
If you have better methods, let me know, thank you.