Can anyone share experiences in terms of needed bandwith to scale with DDP. Is 25GBe enough or do we need 40GBe, 50GBe, 100GBe, 200 GBe? Our target is a small cluster of 4 nodes with 8 GPUs.
Is GPUDirect RDMA a must have for our setup?
What about GPUDirect storage? Is this needed?
Any other considerations we should make to scale well with pytorch DDP?