This has happened with conda installed pytorch=<1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1>, cudatoolkit=<10.1,10.2,11.0,11.1>, with ubuntu18. nvidia-smi shows the driver is version 460.
This happens with K80, T4, and V100 GPUs.
The puzzling bit is the fact that if conda has to download the package, it imports fast; if it uses the cached package, it takes a long time.