Usually most depedndencies like PyBind should be “vendored”, i.e. in git submodules under third_party rather than installed through conda. (use git submodule update --init --recursive or so if you don’t have them).
Notable exceptions are the compute libraries (CuDNN, Magma, MKL, …) and 2-3 Python things numpy, pyyaml and one I forget.
I’m assuming you generally follow the instructions in CONTRIBUTING.md?
Best regards
Thomas