Regarding the concern on speed, hopefully, using more workers in the data loader will help to boost the throughput. We are also working on making the RPC comm layer more efficient by adding TensorPipe 9 as a new backend, so that RPC does not have to do two round trips for each message as with ProcessGroup and TensorPipe would also allow using multiple comm media.