Depending on what you are doing, you might be missing accelerator libraries like MKL or CuDNN.
You should get relatively decent diagnostics what is included from cmake and compare that with what you get when you follow the official instructions for building PyTorch 27 from source.
Indeed,  the official libtorch used to indeed just build with python setup.py and then extracted from the Python build (but I am not sure if that is still the case).
Best regards
Tomas