You can’t build a CUDA extension without the CUDA headers and libraries. I think your main options would be to either require the CUDA toolkit (I don’t think you should actually need a GPU, just the toolkit), or have separate CUDA and non-CUDA versions of your extension.
If going with separate versions you should be able to either have a shared codebase with separate CUDA specific stuff. Or use standard C++ tricks like defines to use the same codebase while excluding CUDA stuff on CPU only builds.
From the python setup.py end you’d want a custom setuptools command 39 to allow custom command line options to control whether you install a CUDAExtension or a CppExtension. Or just have separate python packages.