Yes, you should install at least one system-wide CUDA installation on Windows when you use the GPU package. It’s recommended that you install the same version of CUDA that PyTorch compiles with. It will work even when the two versions mismatch. But you’ll then have to pay attention to the version of the GPU drivers.
Does PyTorch uses it own CUDA or uses the system installed CUDA?
Well, it uses both the local and the system-wide CUDA library on Windows, the system part is nvcuda.dll and nvfatbinaryloader.dll. They are located in the %systemroot%, so I’m afraid we could not put them in the package due to some potential permission issues. The reason why we use the local CUDA installation is to prevent dll load failure. e.g. Some users don’t have CUDNN installed. I don’t think the missing of any optional CUDA dependency should be a barrier to use our package on Windows.
Should pip  install s take care of the CUDA or not?
No, at least the answer is no on Windows. We don’t use pip to install CUDA on Windows. We just use pip to select the package you want. For most users, it may be no differences using cuda80/cuda90. But if you are using some new or legacy cards, then you can only use specific CUDA distributions.