Hey @maralm
From your post, it is unclear which part is the DDP model. My assumption is that:


self.inputs['qa_in'][i]: this is input to DDP forward

self.qa_outputs: this is your DDP model

self.outputs['qa_outputs'][i]: this is your DDP outputs


I think the problem is the self.qa_outputs parameters are used twice in backward but I don’t know how to solve this. I don’t have any problem without distributed.

This should be fine, the autograd engine should be able to manage backward inputs and dependencies from start_loss and end_loss properly.
Two questions:

Does it work if you directly call self.outputs['qa_outputs'][i].sum().backward() after line 3?
Does any of the model parameters or outputs participates in other forward/backward passes?

It will be very helpful for us to debug if you could share a minimum repro example. As we don’t know what happens outside of the posted code snippet, we can only make assumptions.