I had a similar problem while trying to train my network on two GPUs, but I ended up using only one GPU per model and using the other GPU to test out changes in my model and/or parameters.
Since my model was relatively small, I guess i was suffering from the cost of model synchronization mentioned by @albanD.