It works for me, if I just replace the version:
conda install pytorch=1.3.1 cudatoolkit=10.1 -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done
[...]

  added / updated specs:
    - cudatoolkit=10.1
    - pytorch=1.3.1


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    blas-1.0                   |              mkl           6 KB
    cffi-1.14.5                |   py37h261ae71_0         224 KB
    cudatoolkit-10.1.243       |       h6bb024c_0       347.4 MB
    intel-openmp-2020.2        |              254         786 KB
    mkl-2020.2                 |              256       138.3 MB
    mkl-service-2.3.0          |   py37he8ac12f_0          52 KB
    mkl_fft-1.3.0              |   py37h54f3939_0         167 KB
    mkl_random-1.1.1           |   py37h0573a6f_0         322 KB
    ninja-1.10.2               |   py37hff7bd54_0         1.4 MB
    numpy-1.19.2               |   py37h54aff64_0          22 KB
    numpy-base-1.19.2          |   py37hfa32c7d_0         4.1 MB
    pytorch-1.3.1              |py3.7_cuda10.1.243_cudnn7.6.3_0       428.0 MB  pytorch
    six-1.15.0                 |   py37h06a4308_0          27 KB
    ------------------------------------------------------------
                                           Total:       920.9 MB