I don’t think you should do that unless you have multiple GPUs. This 171 might be useful.
My guess is that you are creating as much CUDA contexts as the number of CPU threads, so (i) you allocate too much memory and (ii) the contexts don’t run concurrently.