The problem is that although one can distribute forward-pass and not have it collect on one GPU, there is no way to distribute data across GPUs evenly in DataParallel: the batch goes on GPU0 (or one GPU of your choice), and then that batch get split into further minibatches on other GPUs; as a result GPU0 becomes the memory bottleneck - this article explains it well https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255 29
This behavior of DataParallel isn’t an issue for large models because size(model)>size(batch), but in our case size(model)<<size(batch).