The first arg of rpc_async method specifies the device to run _call_method. Since you used CPU as the device (probably the same core), you actually didn’t parallelize the RPCs.
Can you show the code how you created ag_rrefs? I guess you probably just need to rewrite looping ag_rrefs in this way:
for i in range(len(self.ag_rrefs)):
    ag_rref  = self.ag_rrefs[i]
     # the worker name probably is "worker0" if you only have one machine.
    destination_worker = "{}/cuda:{}".format(worker_name, i) 
    futs.append(rpc_async(destination_worker, ...)

Additionally, unlike rref.local_value() used in _call_method, to move a rref value from a different device, you probably need to call rref.to_here() instead.
Regarding the usage of TensorPipe, please check out this example 1. To support GPU, you have to explicitly specify a device map.