I know if you do not include mkl headers and includes, pytorch builds with eigen to interface with BLAS. Eigen is a lot slower, I would try building libtorch from source using mkl in a conda environment to compare. I am not sure what the prebuilt libs use.