Some optimizers requires memory to work. That’s why I recommend you to use SGD if you have memory issues.  In addition I see that you use 10 times more memory in one gpu.  Check you are not allocating  garbage on the you.
Lastly, I would say you should be able to use a larger batch size. Check that imbalanced usage because it’s not normal.
Can you run even one iteration? Is the memory not being freed? You can you watch -n 0 nvidia-smi to check the memory in real time. If you can run few iterations before getting outnof memory it means you did a mistake in the code.