Hey @heavyfranz
Please see this section in the overview: https://pytorch.org/tutorials/beginner/dist_overview.html#data-parallel-training 6
DataParallel can be slower especially if the model is large, as it will do model replication, input scattering, and output gathering in each forward pass. DistributedDataParallel is expected to be faster in this case.