In theory, yes. As long as you get cards with a higher bandwidth than your Ethernet setup it should result in an improvement. But since NCCL is built for using GPUDirect, I’m not sure if it will work with NCCL out of the box. If it doesn’t, you could try and experiment with IPoIB and fall back to using NCCL’s TCP transport.
Good luck!