bapi:

I am not sure about the topology of the 256 GPUs that you mentioned.


In that test, each node only has 8 GPUs.

So, I figured out that using either the flag NCCL_P2P_LEVEL=0 or NCCL_P2P_DISABLE=1, DDP runs fine on a machine with >8 GPUs.

I see. We don’t have tests covering > 8GPUs per node cases yet. This is an important message, thanks for sharing!