Is it possible to consolidate multiple arr_send[i] into one RPC? Is there any reason that they need to be sent using dedicated RPCs?
Curious:

After 10000s, how many objects are processed?
Is distributed autograd/optimizer also used in this case?
Are any of those CPUs locate on the same machine? (so that shm can be helpful)