I usually recommend to use DistributedDataParallel, as it’s the faster than nn.DataParallel, and also allows you to use SyncBatchNorm layers to share the running stats. I’m not familiar with Horovod, so cannot comment on it.