I would suggest trying something larger in terms of both model and data.
MNIST data is really tiny and your model is likely very small, so you are probably running into more overhead than advantage when using multiple GPUs.
I played around a little but I didn’t actually see much of an advantage unless I moved to large-scale datasets like ImageNet, Pascal etc. where models are larger as well.