Your Python install is not really related to this issue in libtorch.
E.g. I used my base conda environment without PyTorch installed, and could successfully build the C++ example.
Have you built PyTorch from source before?
If so, did you see any issues?
Maybe Peter Goldsborough (one of the PyTorch core devs) will have any idea about this specific issue. CC @goldsborough