Hi @teng-li and @apaszke,
I’m exploring to train a model on a cluster of multiple GPU machines. Each machine has 8 GPU cards. For example, I have 32 GPU cards for training a job on 4 nodes.
The most familiar backend for me is MPI, however, according to How to properly use distributed pytorch with infiniband support, MPI implementation doesn’t support GPU training and GLOO still have some problems.
Can you provide me some advice to train with multiple cards on multiple machines? What is the most reliable backend?