When using GPU, I believe spawn should be used, as according to this multiprocessing best practices 79 page, CUDA context (~500MB) does not fork. This could also be the reason why you see increasing GPU memory footprint when using more spawned processes, as each process will have its dedicated CUDA context.
Curious, can you allocate a different GPU to each different process? Or do they have to use the same GPU in your application?