DDP can solve the imbalanced memory use, but it will use more memory in each GPU for communication, right?
In the case with large sample and large model (e.g. I can only use batch_size=1), is there any suggestion for parallel training?
Thanks a lot!