“did you have the same value of ag_rreff.owner() for different agents)?” -  Yes, the owner is the same, the environment with the CPU.
However, when I create the agents I make them have their own GPU. (Also when I select the action I send the state to their own GPU)
class Agent:
    def __init__(self):
        self.id = rpc.get_worker_info().id
        device = (self.id - 1) % args.num_gpus
        print("Initialasing Agent ID: {}, Device {}".format(self.id, device))
        self.device = ("cuda:" + str(device) if torch.cuda.is_available() else "cpu")
        self.policy = Policy()
        self.policy.to(self.device)
        self.optimizer = optim.Adam(self.policy.parameters(), lr=1e-2)

Does that help??
" I think you can somehow create a mapping and replace ag_rreff.owner() by a cuda device here, even if the environment is in the CPU" - How can I do that? I am a bit of newbie with RPC.
" Another side-suggestion: you can also try TensorPipe RPC backend to accelerate the RPC performance over GPUs, by modifying rpc.init_rpc lines."  - I do not see any GPU options in TensorPipeRpcBackendOptions. Is there something I am missing here?