Could you try to uninstall all previous PyTorch builds and clean the build environment?
conda uninstall pytorch
pip uninstall torch
pip uninstall torch
python setup.py clean  # in your pytorch dir

EDIT: I would also recommend to use a conda virtual environment.