Hi All,
I’m currently submitting a few scripts to a remote server with a few different GPUs and I’ve noticed that in some cases some jobs will fail due to this CUDA unknown error. The GPUs involved are specificied to use CUDA 11.0 and the PyTorch installation is 1.7.1+CUDA11. This error has emerged in few different ways but only when I called CUDA in some way. So, for example, moving my model from CPU to GPU will result in the same error (sometimes). The driver version is 450.66.
Traceback (most recent call last):
  File "~/run.py", line 28, in <module>
    device_name = torch.cuda.get_device_name(torch.cuda.current_device())
  File "~/.local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 366, in current_device
    _lazy_init()
  File "~/.local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 172, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.

Is there any particular way to diagnose the error so that I can resolve it? Thank you!