@mrshenli I am using @rpc.functions.async_execution for the calls from B to A, but I don’t see an easy way to do so for the RPC from A to B.  Remember, internally the RPC from A to B then calls back to B multiple times, so I would need to suspend/resume the thread in order to free it.  (i.e. I would need something like asyncio, which would much further complicate things).
@lcw I am using RRefs, but not dist autograd, or any other advanced features.  I’m not sure how I would transform my code to be non-blocking.  Fundamentally, my RPC running on process B needs to wait until its RPC to process A completes before continuing execution.  I would expect that pytorch distributed should recognize this situation, and return the thread to the thread pool while it is waiting on the RPC.  What could I change in my code so that it “returns the thread” to the thread pool, but continues executing once the RPC completes?