Hey @ryuxin, can this be implemented as a wrapper on top of the RPC API 2? For example, can you implement the queuing logic as an RPC target function? Some related tutorials:

https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html 5
https://github.com/pytorch/tutorials/blob/release/1.6/intermediate_source/rpc_async_execution.rst 5