if you run the setup.py to install Pytorch, then the libtorch will also be built. Here 27 is a little guide on how to build libtorch in a clean anaconda environment. Using it, I also experience that the Python API is ~2x faster. Although it is not as significant, I also wonder where that speed up comes from, since this is the official way to build Pytorch
Python Operation Time(s) 0.0018
C++ Operation Time(s) 0.00416361