To provide more context: I am referring to what is described in pybind11 here: PyBind11-CMake 22
So I figure it should be possible with Pytorch extensions as well and is often favourable over setuptools because it is more platform-independent.
I think @artbataev has done it, am I correct? Maybe you could give me a hint 
Cheers,
Lucas