Machine Learning, Mpi, Pytorch, TensorFlow.
while the runtime schedules operations to be executed in an intelligent order. Theoretically this encompasses all forms of execution. However, the most common kind of parallelism for Deep Learning — data parallel, turned out to be hard to make practical in this form. Early in TensorFlow development