I tried the pytorch-nightly which uses NCCL 2.7.6. I have not faced the deadlock again yet. Thanks @ptrblck.
@Jeffrey_Wang you might want to try that out.
@ptrblck what could be issue with the previous version?