@nathan you may need to invoke two threads each for managing one GPU, because although the CUDA calls are async by default, you might introduce some python-level sync point in your module.
Could you post up a sample code of your submodule?