Thank you for looking into this. I did search a lot before posting here. The closest thing I found was that some of the early RTX 2080Ti releases were defective 41. But in almost all the cases, heating up had lead to an error. It is definitely not PyTorch; most likely the driver or the hardware below it. I figured if anywhere, this community is most likely to see these issues, now, or in the future.
In my situation, these errors don’t happen midst of an experiment when the card is operating at max capacity. It doesn’t heat up; I can’t even get past the first iteration of the training on the RTX 2080Ti anymore. The PSU is 1000W which I believe is more than necessary. No manual overclocking either, not sure if the manufacturer had done anything though.
I’ll update the driver to 415 that is a beta to see if it will fix the issue. I’ll post here if I had any luck.
Thanks.