I was training GCN model on my Linux server and I suddenly got this error.

RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW


Pytorch version: 1.10.1+cu102
OS: Linux
Python version: Python 3.8.10
CUDA Version: 11.2