TensorPipe would fare better if indeed your topology (i.e., the links you actually use) are a significantly smaller subset than the complete graph. (For example, if you use a server-client pattern). In other words, if your graph is sparse. For dense (near-complete) graphs TensorPipe will perform even worse than Gloo because each pipe will internally use multiple TCP connections, whereas Gloo uses only one.
The reason you’re currently unable to use TensorPipe is because, indeed, it uses Gloo internally for the join step. We’ve been wanting to get rid of this for a while but it’s hard, because the RPC agent’s join must do a barrier, and it’s easier to do it through a library that already does collectives (namely Gloo) rather than re-implement it. We could use the c10d::Store instead of the ProcessGroup for that, but currently the Store isn’t powerful enough. @osalpekar was thinking of refactoring it though so maybe then we could do this change. See https://github.com/pytorch/pytorch/issues/42879 2 and https://github.com/pytorch/pytorch/issues/41614 for more context.