You can move to a specific folder to and build it there as described here 4.
Once you’ve built libtorch, you can specify the path with
cmake -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch ..

when building your project, as described here 3.