Another way to do this, which I realized later-on is you can pass the py::object, and then cast to the torch::Dtype (torch::ScalarType):
PYBIND11_MODULE(cytnx,m){
    m.def("func1",[](py::object dtype){
                   torch::ScalarType type = torch::python::detail::py_object_to_dtype(dtype);
             });
}