Hi.
As I mentioned at title, I trained my model in 2 different device environments to compare training speed.
I used torchvision official resnet50 model and CIFAR10 for dataset, which is enough small to run in single GPU.
I found that DDP on 8 GPUs are about 2x slower than single GPU.
Is this expected for small models? Or am I using something wrong with DDP?
+)
I used same parameters for 2 environments, including batch size. Should I incrase batch size for 8 GPUs DDP 8x times? Does this (hopefully always) guarantees similar accuracy with single GPU cases?