What would be the best data-parallel solution regarding the model’s maintaining the same performance or even better compared with training on one GPU?
nn.DataParallel() vs DistributedDataParallel vs PyTorch Lightning Horovod vs any other available methods