When you run model.backward() multiple times without optimizer.zero_grad(), the gradients will be accumulated.
And I think this is just what you want and correct according to your description.
I just wonder why you need to backward twice?
You can just obtain the two outputs and then calculate your loss:
output_a = model(input_a)
output_b = model(input_b)

loss = your_magic_loss_function(output_a, output_b)
optimizer.zero_grad()
loss.backward()
optimizer.step()

You can provide more details about your use-case if you would like to.