What does “sometimes” mean in this context? Is torchvision working in some scripts and raises an error regarding the CUDA version in other scripts?
You should be able to install PyTorch and torchvision using the binary install instructions from here 1 and specifying one CUDA version, which would be compatible with your NVIDIA drivers.