I’ve seen a similar problem when I had an incorrect version of CUDA installed/multiple pytorch installations. I am not sure if the same problem applies to you.
First uninstall all existing versions of torch (pip uninstall torch). It’s possible that multiple torch installations exist in the same conda environment (installed through pip and conda). Ensure that you remove all of them first. If you don’t mind installing torch==1.5.1,  installl pytorch using:

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

The above worked for me in a python 3.6 conda environment.
There are other threads that report the same problem: https://github.com/pytorch/pytorch/issues/6041 8