Solved by ptrblck in post #2 
              
              
                Your local CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension, since the pip wheels and conda binaries use their own CUDA runtime. 
You would thus only need to install a valid NVIDIA driver and can use the binaries directly.