Hi @mrshenli,
Thanks for your reply.
Your assumption is correct and self.qa_outputs is just a linear layer.
Regarding your questions:

No it doesn’t work with that.
No, I am trying to just run a forward layer and compute backward with autograd.backward on that layer instead of running loss.backward().

Basically, I have a large model and when I run it in a conventional way for forward and backward(loss.backward()), it works fine. But, I have a new implementation which runs backward layer by layer using autograd.backward. Using that, the algorithm works fine on a single gpu but face this error in distributed. I tried it on a different model which doesn’t have multiple losses and it is fine. In this case that I add multiple losses, the error comes.