Thanks a lot for your answer.
Compute things once before the loop helps a lot for the memory.
I didn’t want to add temporary variable for memory concerns, but it was worse since it should have kept copies to compute backpropagation.