This will be more of an open question, but I'm hoping someone may know how to solve my problem.
I have C++ library compiled with pybind11, allowing me to call some functions from python. The C++ library uses CUDA to perform some operations on the device.
I need to allow the user (who only has access to python, and cannot go into the C++ or CUDA code) to define an operation (typically, a lambda) that will run in one of the CUDA kernels.
If you have any pointers as to where I could learn the proper way of doing this, I'd be very grateful!
I can't think any way to do this. Barriers to making something like this work

PyBind style bindings won't work with CUDA device functions
While CUDA supports passing of C++ lambda functions to kernels, it is a static compiler generated support. There is no way I am aware of to get a lambda function into a kernel call without compiler invocation (and probably nvcc and not nvrtc, which would imply a lot of messing around with compilers at runtime, or hacking extensions to PyCUDA). All of that would violate your "cannot go into the C++ or CUDA code" criteria.
Similarly, you would require CUDA runtime linking support in your application. I'm also pretty certain that the linker can't link naked lambda functions, only device functions with a generic interface (nvstd::function), so you would need to emit kernels, wrappers, and CUDA C++ device lambdas, compile them, and link them. That requires the CUDA driver API which I am sure you are not using.

The best hope you have would be to try Numba, which allows runtime compiling of Python device functions. However it lacks any ability to deal with lambda functions and can't interact with C++ code statically or JIT compiled with the CUDA toolchain. You might be forced to reimplement a lot of your CUDA C++ code in Python to make that work, and the Numba kernel dialect lacks a lot of modern CUDA features, which might make that extremely difficult depending on how sophisticated your existing CUDA code is.
Best way to pass a lambda to a CUDA kernel from python code