Machine Learning, Mpi, Pytorch, TensorFlow.
can program an arbitrary program scheduler as a TensorFlow graph, like was done in Loom, but it’s unintuitive and will probably be never repeated again. Typically you are limited to standard model-parallelism/data parallelism patterns and hope that scheduler will do the right thing. Running