William_Ravenscroft:

RuntimeError: NCCL Error 3: internal error



NCCL error 3 seems to be either a bug in NCCL or some memory corruption: Types — NCCL 2.8.3 documentation 7. Maybe you can create an issue at GitHub - NVIDIA/nccl: Optimized primitives for collective multi-GPU communication 12 to see if the NCCL team has some guidelines on how to debug this.