Thanks for your input. Certainly interesting that you can see scalability for 512 GPUs - We are looking at really large scale and 5000 GPUs would be the lower bound. Some facts about the hardware:

IBM Power9 with Nvidia Volta V100 GPUs
GPUs have NVLink (Gen 2) interconnect, with 4 GPUs/node.
Mellanox EDR Infiniband.

I want to train a custom architecture which is similar to a convolutional recurrent network - we will be training on several terabytes of training data. Apart from the inherent issues of communication latency etc., is there any limitation in Pytorch as such when I scale up?
I am having difficulty finding resources to learn distributed GPU training with Pytorch (apart from documentation) - What approach  did you use? I am currently experimenting with Horovod on a much smaller scale, but I’d prefer using Pytorch’s native distributed module, if I can only get some nice tutorials to study. Thanks again for your insights.