Scaling Kubernetes to 7,500 Nodes
><i>Pods communicate directly with one another on their pod IP addresses with MPI via SSH, not service endpoints.</i><p>this is super interesting. They are using mpi on kubernetes for their AI training i suppose.<p>So they are not using anything like kubeflow. Any idea which framework this is ?<p>The state of AI training on kubernetes is not so hot. And this would be a good learning. There is Ray Distributed today that claims a better performance (as well as a better developer experience) than OpenMPI - <a href="https://www.usenix.org/system/files/osdi18-moritz.pdf" rel="nofollow">https://www.usenix.org/system/files/osdi18-moritz.pdf</a><p>wonder why the choices were made as such