@BH4CYI Might be useful to directly contact the NCCL team here to debug this further. I’d suggest probably creating a gh issue here: GitHub - NVIDIA/nccl: Optimized primitives for collective multi-GPU communication 2 and sharing all the NCCL debug logs with them to determine a resolution.