Well, if you are using GPU, then you are still doing computations on the same GPU, but on CPU, you are…running some kind of multiprocessing parallel.
Please read multiprocessing tutorials first.
This question belongs to distributed and not RL