The training might differ generally, as the gradients will be reduced from all devices to the default device, which would be equivalent to training with a bigger batch size.
Also, if you are using batchnorm layers, their running estimates won’t be synchronized in nn.DataParallel, which might also make a difference.
We recommend to use DistributedDataParallel with a single process per GPU for the best performance. DDP also supports SyncBatchNorm, which might be beneficial.