Thanks a lot for this pointer, it’s really helpful. Btw, I noticed that MPI initialization part in the doc of distributed pytorch is kind of “missing”. Do you need someone to help with that? If so, I can do it.