Thanks a lot, ptrblck. But the memory usage still increased crazy even with the second method(maybe the problem is that I set create_graph=True in autograd), I just wondering why the memory usage do not increase when the gradient calculation is inside the loop? And also when do I need create_graph=True?