The fastest way probably from the point where you are at is to just use the LibTorch from the PyTorch you built. It contains all you need in torch/lib, torch/shared, torch/include from the Python installed directory (/usr/local/lib/python*/dist-packages/torch/* for me, try torch.__file__ if you can’t find it).
Best regards
Thomas