@mrshenli Yes, that seems to fix the issue.  However, I am still puzzled about the original behavior.  Running out of worker threads should cause the execution of the RPC to be blocked until the threads are freed, but it seems like instead it is breaking other things.
This sounds like a bug, no?
To simplify my setup, I basically have two processes, process A and process B.  Process A is sending 16 RPCs to process B, and all these RPCs are completely independent of each other.  Each RPC from A->B internally sends a sequence of RPCs back to A.  I don’t see why this should cause a deadlock, even if the thread pool in each process is size 1, because only 1 simultaneous RPC per process is required to make forward progress.
My guess is that the thread pool is being reused to both send and receive RPCs, so all 16 threads are taken by the RPCs from A to B, but now all these threads are stuck, because they are unable to make RPCs from B to A?