If you only need the CPU version you can specify it :
conda install pytorch torchvision cpuonly -c pytorch


or
pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html