It turns out using CUDA 11.2 instead of 11.3 solved my problem.
Thanks for your reply @ptrblck. Updating the drivers on the bare metal might work too, but I am happy to just let debian’s APT handle the drivers on the bare metal node for now.