Yes, though your method1 is bad, it is better to accumulate gradients with loss.backward() in a loop, this releases “backward graphs”. But you should use method2, unless there are obstacles with some unimplemented batched operations.