Your install method is the reason why it’s so slow.
You have two options:

Install the binary from pytorch.org 23 (we provide wheel files and conda binaries)
Install a proper BLAS library on your machine (MKL or OpenBLAS, MKL is available via Conda) and add it to the environment variable CMAKE_PREFIX_PATH. Otherwise pytorch will use super-unoptimized BLAS, and that’s the reason for the slowness. You can read one way of doing this here: https://github.com/pytorch/pytorch#from-source 57