For more on Cray&#x27;s Aries interconnect used on the Cray XC-30 referred to in the OP, see this paper from NERSC, which has one: <a href="https:&#x2F;&#x2F;www.nersc.gov&#x2F;assets&#x2F;pubs_presos&#x2F;NUG2014Aries.pdf" rel="nofollow">https:&#x2F;&#x2F;www.nersc.gov&#x2F;assets&#x2F;pubs_presos&#x2F;NUG2014Aries.pdf</a><p>Depending on topology, you can get ~1 µs latency with Aries.  It&#x27;s based on PCIe-3, so it&#x27;s different from InfiniBand.