Yeah, that is the wrong prefix path.
I always use
Torch_DIR=/Desktop/sarvagya/project_folder/env/lib/python3.7/site-packages/torch/share/cmake cmake ..
and that seems to work for me (well, I have a different path to python3.7, but the general idea).