Well, it depends on several things. I found that some optimizers requires lot of memory. I got this issue using adam. Those optimizers save their parameters in by-default gpu. I’m trying to figure out if there is a way of solving that problem, however it seems there is no simple way to store those parameters in cpu or another gpu.
At the same time, when you load input variables they have to be saved in gpu. You can try to save some memory allocating the input in a gpu different from by-default one.
Anyone you can try to use a simpler optimizer such us SGD.