I assume you’ve build PyTorch in the ~/pytorch folder and installed it via pip or conda?
In that case, could you try to use:
cmake -DCMAKE_PREFIX_PATH=`python -c 'import torch;print(torch.utils.cmake_prefix_path)'`

instead of the absolute path to the downloaded libtorch?
The small Python inline script should return the path to your locally built cmake files.