pip also downloads the binaries for cuda and cudnn. You only need to install cuda and cudnn in your system if you want to compile from source.
Anyway I recommend to use conda. Previously I used Pip until I realized that on some versions numpy was not downloaded. Pytorch documentation recommends to use conda.