Thanks @rasbt!
Reading your answer gave me an idea to search if there was a torchvision package for CPU since this one it seems like it was for GPU. And I found it! 13
You only have to do:
conda install -c pytorch torchvision-cpu
And it comes with:
The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pytorch-cpu-1.0.1          |      py3.7_cpu_2        26.8 MB  pytorch
    torchvision-cpu-0.2.2      |             py_3          44 KB  pytorch
    ------------------------------------------------------------
                                           Total:        26.9 MB

The only drawback is that it comes with an earlier version of Pytorch than the specified on repository I am working, but I don’t think it would be that much different.
So if anybody has this exact question in the future here is an answer, I hope it helps!