The PyTorch binaries will ship with the CUDA, cudnn, etc. runtime libs, so that you can use it in PyTorch directly.
The binaries will not install the complete CUDA toolkit on your machine with the compiler.