Your main problem is you are using Adam. Adam parameters take memory from the main gpu to store its parameters. As far as I know there is nothing  you can do. The bigger your model is, the worse this problem will be.
You can try to ask some developer if there is a way of spliting the memory usage of adam among all the gpus but, as you could see in the thread i opened, I got no answer.