Did you build PyTorch from source?
If so, could you try to clean the build (python setup.py clean) and uninstall all old versions?
pip uninstall torch
pip uninstall torch
conda uninstall pytorch

(also *-nighly builds if previously installed).
However, if you didn’t build it from source, you don’t need to install CUDA on your system, as the PyTorch binaries will already ship with the CUDA and cudnn runtime libs.