I use nvidia-smi to watch the status of gpu while running programs. There are two possible cases:

the nvidia-smi will fail to obtain the status of gpus, while programs are still running. The error says you need to reboot your computer. After a while,
the computer was totally stuck and cannot be ssh-ed.

The temperature is normal even at the point when it gets stuck.