Faced a similar error - solved it by initializing the process group first, and then setting the model cuda device (as opposed to the other way around, which led to the same kind of deadlock you describe)