I got around to running a test with the Gloo benchmark tool and confirmed the issue.
For a larger explanation of the issue and the fix, see https://github.com/facebookincubator/gloo/pull/192 27.
Once this is merged and bumped in PyTorch, I expect you’ll be able to run the same test and find the bandwidth to be very close to link speed.