If I have 4 GPUs, is that possible to do the follow:
Use one GPU to store intermediate outputs and aggregates them together to backprop. Mount model and minibatch to the rest 3 GPUs calculating forward.
In this case, I’m able to train larger model with larger batch size. Currently, the ability is bounded by GPU memory as Pytorch always gather all outputs to one GPU which also is responsible for forward() calculation.