Sure. It seems to be PyTorch related because the other GPU usages are fine.
My dmesg is full of:
[  959.195648] pcieport 0000:00:03.0: AER: Multiple Corrected error received: 0000:00:03.0
[  959.195768] pcieport 0000:00:03.0: AER: can't find device of ID0018
[  959.195769] pcieport 0000:00:03.0: AER: Multiple Corrected error received: 0000:00:03.0
[  959.195898] pcieport 0000:00:03.0: AER: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[  959.195902] pcieport 0000:00:03.0: AER:   device [8086:6f08] error status/mask=00001040/00002000
[  959.195903] pcieport 0000:00:03.0: AER:    [ 6] BadTLP                
[  959.195908] pcieport 0000:00:03.0: AER:    [12] Timeout               
[  959.195915] pcieport 0000:00:03.0: AER: Multiple Corrected error received: 0000:00:03.0
[  959.195986] pcieport 0000:00:03.0: AER: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[  959.195987] pcieport 0000:00:03.0: AER:   device [8086:6f08] error status/mask=00000040/00002000
[  959.195989] pcieport 0000:00:03.0: AER:    [ 6] BadTLP                
[  959.195992] pcieport 0000:00:03.0: AER: Corrected error received: 0000:00:03.0
[  959.195996] pcieport 0000:00:03.0: AER: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[  959.195998] pcieport 0000:00:03.0: AER:   device [8086:6f08] error status/mask=00000040/00002000
[  959.196000] pcieport 0000:00:03.0: AER:    [ 6] BadTLP                
[  959.196007] pcieport 0000:00:03.0: AER: Corrected error received: 0000:00:03.0
[  959.196148] pcieport 0000:00:03.0: AER: can't find device of ID0018
[  959.196150] pcieport 0000:00:03.0: AER: Multiple Corrected error received: 0000:00:03.0
[  959.196291] pcieport 0000:00:03.0: AER: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
[  959.196293] pcieport 0000:00:03.0: AER:   device [8086:6f08] error status/mask=00000040/00002000
[  959.196294] pcieport 0000:00:03.0: AER:    [ 6] BadTLP                
[  959.196300] pcieport 0000:00:03.0: AER: Multiple Corrected error received: 0000:00:03.0

-Edit: When it breaks I get a lot of EDAC sbridge: Seeking for: PCI ID 8086:6f6d
I also noticed that the training runs well for a while and then when the errors start to appear the time to finish the epoch suddenly increases a lot until eventually the program freezes.