Have some updates.
I tried uninstalling all my cuda stuff and installed cuda 10.2 from scratch. I managed to build pytorch with cuda 10.2. This is where I discovered the following behavior:




import torch
torch.randn(1, device=“cuda”)
Traceback (most recent call last):
File “”, line 1, in 
RuntimeError: CUDA error: no kernel image is available for execution on the device




x = torch.randn(1024, 1024).cuda()
print(x.device)
cuda:0




y = torch.matmul(x, x)
Traceback (most recent call last):
File “”, line 1, in 
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling cublasCreate(handle)




So I think the fact I am now using cuda 10.2 has nothing to do with this behaviour. I think it also behaved this way before, when I was on cuda 11.2.
Does this info point in any direction?