Could you check, if the memory usage increases in each iteration if you are using retain_graph=True? If so, this should also show a slowdown in each iteration and might be the reason for the huge performance difference.
I’m not familiar with your specific use case, but are you seeing an error by removing retain_graph=True?