The PyTorch binaries ship with their own CUDA runtime, so you would only need to install the appropriate NVIDIA driver to run PyTorch workloads. Your local CUDA toolkit will be used, if you want to build a custom CUDA extension or PyTorch from source.