You could try to add CUDA streams as described here, but would have to take care of the synchronizations to avoid race conditions.
I would recommend to try it out with a “similar” but simple model and check, if you would get any performance gains at all.