When you compute the forward pass using different inputs, each output will have its own computation graph attached to it. It’s not overwritten by the next call, so there should be no problem in your current approach.
You are right that the gradients are accumulated in consecutive backward calls, but that doesn’t seem to be the case in your example.