I guess it could be the PSU. I figured the stability when using other libraries counted against this theory. It’s supposed to deliver 1000W, which should be plenty for a single GPU system. I’m not seeing XIDs in the logs, or really any reoccurring message preceding the crashes. (This is when looking at /var/log/kern.log and output from journalctl in Ubuntu 18.04.)
I’ve discovered that setting num_workers=0 on the PyTorch DataLoader makes things considerably more stable (although crashes can still happen after several hours). All four code bases that have crashed have relied on the PyTorch DataLoader. Of course, setting num_workers=0 also slows down execution, causing less stress on the GPU, power draw, etc, so I assume that is why this helps. I don’t really see how a DataLoader bug could bring down the entire system in any case.
It’s possible that the four PyTorch code bases just happen to be able to saturate the hardware better, thus causing the crash, for example through power draw. I suppose I should try very hard to get things to crash with other libraries, which would rule out PyTorch altogether as a part of the problem.