But I can’t tell where is the part related to using CPU GLOO instead of GPU GLOO in this initialization. Do you mean when creating the model, just like what we do without DDP, don’t use model.to(device) to use make the model run on CPU?

https://pytorch.org/docs/stable/distributed.html
" torch.distributed supports three built-in backends, each with different capabilities. The table below shows which functions are available for use with CPU / CUDA tensors."
You simply initialize the backend.  The model is on CPU unless moved to GPU.

It seems like the device_ids argument is also not needed in DDP initialization, right?

https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html 1
" * device_ids (list of python:int or torch.device) –CUDA devices. 1) For single-device modules, device_ids can contain exactly one device id, which represents the only CUDA device where the input module corresponding to this process resides. Alternatively, device_ids can also be None. 2) For multi-device modules and CPU modules, device_ids must be None.When device_ids is None for both cases, both the input data for the forward pass and the actual module must be placed on the correct device. (default: None)"