All errors occur ONLY on the “bad” GPU. Above I just want to clarify that even with the latest stable releases of pytorch, errors still occur on the “bad” GPU.


I just ran the scripts as it is. In this situation, even with the “bad” GPU, the main.py in pytorch/mnist runs well; no error occurs. But the main.py in pytorch/word_language_model (using the LSTM model) will have the error “RuntimeError: CUDA error: an illegal memory access was encountered”.


In the second situation where I made a minimal change to the scripts for the testing purpose: I added the line “torch.autograd.set_detect_anomaly(True):” prior to the iteration for loop. Then, in BOTH scripts of mnist and word_language_model, appears the error " RuntimeError: Function ‘TBackward’ returned nan values in its 0th output" (not necessarily the TBackward, but could be other functions).