They do the same, but with DataParallel you drive N devices from a single process, whereas with DistributedDataParallel you drive N devices from N processes. For the latter, those devices may be located in different machines, hence the distributed part. You’ll have to launch those N processes though, you cannot start a single process and have it work out of the box. You can start multiple processes manually (and set the RANK environment variable accordingly, per the error message you’re seeing), or use the torch.distributed.launch 6 utility to launch processes for you.