Thanks for your reply, let me make it clearer.
I have two stages of training. The first stage needs 1 image which works fine. The second phase, transfer, needs to feed two images, but runing out of memory.
What makes me confused is that, a single GPU can handle 1 image and the entire network, but 3 GPUs cannot handle 2 images and only the backbone. And after I splitted the first 30 layers of VGG16 into 3 GPUs, the second part consisting of 5 layers was where the model ran out of memory, rather than the bigger part 1 or part 3.
And one more thing, during transfer (stage 2), I call loss.backward() every 4 batches (8 images totally) and add up the losses before calling to gather enough features for my training. Don’t know if this has something to do with the issue.