I’n trying to train models with 4 gpus(1080ti) with batch of 128, but got out of memory problem.
The gpu0 is actually out of memory, and other 3 gpus has 7G out of 12G which is very sufficient.
The main reason is that all inputs has to be on gpu 0 first, and it is replicated to other gpus later.
It would be much efficient of inputs can be run in parallel. Is it possible?