I just tried running

conda install pytorch torchvision torchaudio cudatoolkit=10.1 -c pytorch

and it worked. It upgraded my cuda to 1.4,

The following packages will be UPDATED:
pytorch            0.3.0-py27_cuda8.0.61_cudnn7.0.3hf383~ --> 1.4.0-py2.7_cuda10.1.243_cudnn7.6.3_0
The following packages will be DOWNGRADED:
cudatoolkit                            10.2.89-hfd86e86_1 --> 10.1.243-h6bb024c_0

Do you think this is new enough?
(the errors I posted above came from running

conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch

I guess conda somehow decides that I can’t run cuda-11.1, so it gave me a cpu-only version.
But I already had cuda-11.1 installed. Maybe it is because I am recently upgraded my OS to Ubuntu 20.04?)