jeremysalwen:

My guess is that the thread pool is being reused to both send and receive RPCs, so all 16 threads are taken by the RPCs from A to B, but now all these threads are stuck, because they are unable to make RPCs from B to A?


This is true for temporary ProcessGroup backend, but I am not 100% if this (send/recv share the same thread pool) is the case for TensorPipe backend. @lcw could you please confirm?
The default RPC functions will block one thread on the callee side until the response is ready. If you would like the response being processed asynchronously, you can decorate the user function with @rpc.functions.async_execution and let the user function return a Future. This should release the thread on callee as soon as it gets the Future object, and can resume running the callback on the Future when the Future is completed.