What I would like to do is to have an agent per GPU and then the environment in the CPU.

                rpc_async(
                    ag_rreff.owner(),
                    _call_method,
                    args=(Agent.select_action, ag_rreff, state, step)
                )

To have each GPU dedicated to an agent, I think you can somehow create a mapping and replace ag_rreff.owner() by a cuda device here, even if the environment is in the CPU. Did you assign different agents to the same CPU (i.e., did you have the same value of ag_rreff.owner() for different agents)?
Another side-suggestion: you can also try TensorPipe RPC backend to accelerate the RPC performance over GPUs, by modifying rpc.init_rpc lines.