Maybe. The other day, one of the GPU in the server does not show up when executing nvidia-smi. The root user has reboot the computer once. After that, I start training without error until I ran into this issue.
How to check if the GPU is overheating?