Hey @ZiyiZhu  Are you trying to run this with RPC? Currently init_rpc does not work together with init_process_group. There are work around to create non-default process groups. Or we can also add a fix to init_rpc if necessary. This is the tracking issue: https://github.com/pytorch/pytorch/issues/33583 16