In general we recommend using DistributedDataParallel over DataParallel. Yes, the setup for DistributedDataParallel is indeed different from DataParallel since it requires setting up process group and also multiple processes. My recommendation would be to just have one version of your training script and only use DistributedDataParallel.