I think using an appropriate CMake + make install should work, e.g. the Android build does this. You want to disable the Python bit for this.
The suggested alternative there works well, too - building Python and picking the lib and include. Libtorch 1.0 used to be built that way (actually extracting from the whl…).
Best regards
Thomas