Did you play around with some hyperparameters, e.g. did you try to lower the learning rate for the multi-GPU setup?
I assume your model doesn’t converge that well using 4 GPUs compared to the 2GPU run?