Not sure. Maybe because if you keep the batch size and use multiple GPUs, then each model will only get 1/num_gpus of the data points and it’s then noisier when the gradients are combined from the 4 different models to update the model for the next round. And if you increate the batch size by num_gpus such that each GPU has the same batch size as in 1-GPU mode, the averaging over the gradients may also not be ideal because you loose too much signal from that.
Maybe because I am a scientist not an engineer, but I am generally not a big fan of the parallel approaches for DL because I like the method to be agnostic of what device and how many devices is/are being used