Rerun your script with NCCL_DEBUG=INFO python script.pt args and check what causes the system error. The first error messages complaining about the device were caused as you didn’t push the parameters to the device via .cuda().