Double post from here 17 with a potential answer.
Cross-posting:

you don’t need a local CUDA toolkit installation if you don’t plan on building PyTorch from source or any CUDA extension, just the NVIDIA driver.
the binaries use CUDA11.0.3, so 11.0 Update 1. If you plan on building custom CUDA extensions, you should install the same local toolkit