@Frazer You do not need to change the device_ids  argument passed to DDP in the distributed tutorial even if you use 2 GPUs. The demo_basic function is run on each spawned process, so each process ends up with a replica of ToyModel wrapped by DDP.