Mriganka_Nath:


For the optimizer part, I am using 2 optimizers, so should I use optimizer.step() and optimizer.zero_grad() after calling loss.backward() for each model or do it in the end.



I don’t think it matters much as long as for each model you keep the
zero_grads → backward → step order.



When we do this input.requires_grad_() what’s the difference between input.requires_grad = True



Mainly the looks and that it throws an error when you misspell requires_grad_ rather than assigning some useless field.



like suppose in the 2nd epoch, if the input.grad is still on, so if we find grad again wont it be grad of grads .



The typical thing is that you get (batches of) your inputs from a dataloader. In this case the tensors with the inputs are re-created in every epoch, so no need to worry about it.



Can you explain more why we are doing this

if input.grad is not None:
    with torch.no_grad():
        input.grad.zero_()



I don’t think you need this in most cases (you could add a print statement to see if this ever is the case).
If you were to re-use inputs for whatever reason, this does the equivalent of optim.zero_grad() but for the input. (Because it isn’t in any optimizer.)
Best regards
Thomas