I do not have Nvidia GPU on my own to make sure about solutions.
Could you try this:
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

But before this make sure you have uninstalled any other previous versions.