This topic is old but I just sliced my toes off with a similar issue at head and thought I’d document what I found. I had a similar use case (pybind11 method in a project that depends on PyTorch needing to accept a torch::jit::StrongFunctionPtr and being somewhat unsuccessful at it in certain situations).
Recent PyTorch wheels do include pybind11 headers corresponding to the submodule, afaict, and I was using a workaround in my build setup to always include PyTorch headers at the front of the include path for the parts of the project that depend on pybind11. Not great, but worked, and I can’t think of a better way to depend on a binary distribution like that.
However, when I built from source, my installed package did not include the headers. I believe this is because of detection logic choosing to use the version of pybind11 that I pip installed (i.e. pip install pybind11). In my situation, I also had a system package for pybind installed at an old version. I spent several hours untangling which were used where and systematizing it.
It would be really handy if both PyTorch and pybind11 stamped the actual version or commit hash used somewhere in their distribution. Maybe even a symbol in the resultant binaries that carried the version in its name somehow (so it could be inspected with nm). In the absence of that, some better documentation of how this dependency is resolved would help a lot.
It could also work if we all just uniformly switched to the pip installed versions and included the right version spec in our requirements.txt file. That would probably break a lot of people, though, who don’t really need cross-module coordination on types.