Sorry, I’m not really familiar with distributed training and gloo, so I can’t give any useful input to fix this issue. 
However, have you thought about other approaches to avoid the OOM issues?
torch.utils.checkpoint 9 might be worth a try (although I’m not sure how it behaves in a distributed setup) or NVIDIA’s apex - mixed precision training 6.
Let me know, this would be an option.