Apart from the suggestions here,  did you try to increase the learning rate because you increased the batch size? In your single gpu usecase,  the number of gradient steps is 4x than multi gpu use case. But higher batch size might give a better approximated gradient. So you can very well increase the learning rate and see if the accuracy improves.