I was wondering if anyone can point me to resources on how Pytorch scales on extremely large scale parallel jobs - I am researcher with a group working on ML for computational physics, with all my codes being in Pytorch.
Currently we are evaluating ML frameworks which can scale on super-massive distributed jobs i.e. > 5000 GPUs.  Any idea on what is the largest distributed job using Pytorch so far (both in public domain and internally at Facebook) ?  Are there any fundamental limitations which prevent it from scaling at that level? If so, can I expect any improvements with 1.0 release?
I would like to avoid re-writing all my codes in Tensorflow for scaling, so I’d really appreciate some input. Thanks in advance…