Thanks for quick reply.



 gcramer23:

Yes, initialize a process group before creating your DDP model.


I have indeed followed the link you metioned. And to initialize a process group, I used the code:
def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # initialize the process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)

But I can’t tell where is the part related to using CPU GLOO instead of GPU GLOO in this initialization. Do you mean when creating the model, just like what we do without DDP, don’t use model.to(device) to use make the model run on CPU?
For example:
    setup(rank, world_size)

    # create model and move it to CPU? with id rank
    model = ToyModel()
    ddp_model = DDP(model)

It seems like the device_ids argument is also not needed in DDP initialization, right?