Hey all, I have encountered the same issue on a dual GPU. I am using Pytorch 1.0 and have experienced random crashes. One observation as @Sn_T noted that if I increase the number of batches more than what the total GPU memory allows, it definitely crashes instead of throwing a cuda out of memory error.
Other times I have experienced crashes after having to force terminate the training mid way and restart. Just a side note than num_workers = 8
Was a faulty PSU really the problem? I am using “700 Watt be quiet! Pure Power 10 CM Modular 80+”
https://www.mindfactory.de/product_info.php/700-Watt-be-quiet--Pure-Power-10-CM-Modular-80--Silver_1138276.html 7
Anyone found the source of crash? Thanks in advance.