So I’m still running into this. For additional context, I’m using PytorchLightning with distributed data parallel on 6 GPUs. Training proceeds normally for a few epochs, and then crashes with the error given above. I also just verified that the same things happens when using a single GPU.
@ptrblck  I’m not sure what I’m looking for in dmesg. dmesg | grep xid comes up empty.