Hi,
I am using DistributedDataParallel with nccl. I have two losses which are averaged before calling backward. But backward doesn’t work. Here is the part of the code that is problematic:
self.inputs['qa_in'][i] = Variable (self.inputs['qa_in'][i].data, requires_grad=True)

self.outputs['qa_outputs'][i] = self.qa_outputs(self.inputs['qa_in'][i])

start_logits, end_logits = self.outputs['qa_outputs'][i].split(1, dim=-1)
start_logits = start_logits.squeeze(-1)
end_logits = end_logits.squeeze(-1)

ignored_index = start_logits.size(1)
        
start_positions_ubatches[i].clamp_(0, ignored_index)
end_positions_ubatches[i].clamp_(0, ignored_index)

loss_fct = CrossEntropyLoss(ignore_index=ignored_index)

start_loss = loss_fct(start_logits, start_positions_ubatches[i])
end_loss = loss_fct(end_logits, end_positions_ubatches[i])

self.outputs['loss_out'][i] = (start_loss + end_loss) / 2
self.outputs['loss_out'][i].backward( retain_graph=True)

and I get the following error:
File “/home/suncast/venv3/lib/python3.6/site-packages/torch/autograd/init.py”, line 99, in backward
allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Expected to mark a variable ready only once. This error is caused by use of a module parameter outside the forward function. The return value of the forward function is inspected by the distributed data parallel wrapper to figure out if any of the module’s parameters went unused. If this is the case, it knows they won’t receive gradients in a backward pass. If any of those parameters are then used outside forward, this error condition is triggered. You can disable unused parameter detection by passing the keyword argument find_unused_parameters=False to torch.nn.parallel.DistributedDataParallel. (mark_variable_ready at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:342)
I think the problem is the self.qa_outputs parameters are used twice in backward but I don’t know how to solve this. I don’t have any problem without distributed.