I think you are right, so I just went to the computer room to disassemble my computer. I tried to clean and reapply the silicone grease for my two problematic devices.
Then I found that after removing the two GPUs that were out of order, the two GPUs that I used to work normally also had problems.

Device 0（Before it was device 1）raise RuntimeError: CUDA error: unspecified launch failure

Device 1（Before it was device 3）raise cuDNN error: CUDNN_STATUS_INTERNAL_ERROR


They seem to inherit the original bug.

QQ图片20211105172016714×289 13.5 KB

This device should be the original device 3. I make sure that I just started the computer, and their temperature is not high. I am a little confused now.