In my experience GPUs don’t multitask very efficiently. It’s usually better to run jobs on non-overlapping GPUs. (e.g. use CUDA_VISIBLE_DEVICES). Often, CUDA kernels are written to take advantage of the entire GPU.
You may still get some speed-up by, say running 2 jobs per GPU, but probably not 2x. Of course, this is dependent on your problem.
If you try multiple jobs per GPU, please report back what you see. I’m curious to know.