Not professional but in my opinion, what your method 1 doing is: ‘compute grad 1 → optimize along grad 1 → compute grad 2 → optimize along grad 2’, so when the second forward is performed the model params are already updated once, whereas in your method 2, it’s ‘compute grad 1 → compute and add grad 2 to grad 1 to form grad_total → optimize along grad_total’, there’s only one optimization step going towards the integrated gradient instead of two separated steps. I think this difference is like when you have two dataset, method 1 is alternatively feeding a batch of one dataset at a time and method 2 is mixing two batches one for each dataset and feeding them together.
As for why backward twice saves memory, because once the loss is backward, memories of tensors involved in computing its forward step are freed (that’s why you cannot backward twice with one forward) whereas if you were to accumulate the loss and backward together, the memories are still retained when performing the second forward step.