Machine Learning, Mpi, Pytorch, TensorFlow.
it was discovered that sometimes graphs were too big. Take a common use-case — asynchronous gradient descent with parameter server and 50 replicas. Combine that with inefficient representation of the model, and you end up communicating 100GB graph description to your workers. As a result, an