I got 4GPUs so I let the (batch size)4 together with the LR3.69, but the accuracy of the model seems to get decreased. Must the LR multi the same times? And what if I used batch size with 3 times of the original but train on 4 GPUs, will this influence the performance?
Final question, what would be the best data-parallel solution regarding the model’s maintaining the same performance or even better compared with training on one GPU?
nn.DataParallel() vs DistributedDataParallel vs PyTorch Lightning Horovod