I am not sure if the multiprocessing will only paralize the cpu or both cpu and gpu?
According to the official documentation of Process Pools:
processes is the number of worker processes to use. If processes is None then the number returned by os.cpu_count() is used.
Therefore, I believe it should only do the parallelization on CPU.

There is no need to communicate across each process?

I think even if you parallelize the loops here, you will still have to sync gradients/weights across processes, because there is actually an implicit data dependency of the model weights.
If you parallelize the cross validation for loop w/o communication, you will evolve a separate models on each process, and this will slower down the convergence.