I was getting the same on Ubuntu 18.04 with CUDA 9.1. I found that  setting up a new environment, with
conda install pytorch torchvision cuda91 -c pytorch
rather than letting it sort out it’s own dependancies.
Worked for me.