Thank you for your response.
After some testing, now I realised that the hang is due to syncBNs in the model - it works fine with normal BNs. The graph between input2 and input1 has syncBNs too and the many autograd.grad() calls give rise to many all_reduce calls in syncBNs’ backward which hang. I think this is what’s happening - https://github.com/pytorch/pytorch/pull/14267#discussion_r257051495 16
My model has many heads in parallel with syncBNs and those could be deadlocking too.
Do you see a solution/workaround for this?
Thanks again!