Hi Ignacio-Rocco, I just met the exactly same problem and in my case it seems happen only occasionally, and sometimes there is no such problem. I wonder if you could explain a little more on the point “recompiling pytorch with the ‘included’ NCCL”. What do you mean by ‘included NCCL’?
I am using torch 1.4.0 .
Thanks a lot.
EDIT: The reason is setting somewhere os.environment['CUDA_LAUNCH_BLOCKING'] = "1" .