You can also use your build directly without going through conda:
mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH=<whatever>/pytorch/torch/share/cmake ..
make

Where the pytorch is the top level folder you have the PyTorch repo checked out to (e.g. the folder produced when you run git clone https://github.com/pytorch/pytorch.git