I also prefer it most of the time. What I meant though was that you were calling
pip install torch torchvision` 

in your example above that failed instead of
conda install pytorch torchvision -c pytorch

(you should maybe try the latter to see if it solves your problem)